Okay I figured it out. Pretty silly actually...
Okay I figured it out. Its pretty silly actually. You see, I never restarted linux after after disabling AGP. Everytime I rebooted (and fsck'd) after each crash, I put the AGP option back to default, before launching X again. This because I somehow believed that disabling AGP in my XF86Config-4 was the cause of my problem.
It was actually because the AGPGART driver was loaded in the kernel. Whenever X starts and AGP is enabled, it loads the AGPGART kernel module if it isn't already. When I disabled AGP in XF86Config-4 and ran X again (without rebooting), AGPGART was still loaded into the kernel. My guess is that AGPGART module some how changes they way AGP Bus behaves. When disabled, the AGP bus should act like a PCI bus. Perhaps the AGPGART module changes the bus' behavior and causes X to crash when it expects it to behave like a PCI bus. Again, thats just my guess. Please feel free to correct me.