Well, it could be the X and AGP configuration in the kernel.
I used to build the kernel with AGPGART support, but then switched to nvidia's AGP driver hoping it would solve my lockup problems. It didn't, but I never bothered to change the config back.
# Option "NvAGP" "0" # disable AGP
Option "NvAGP" "1" # use NvAGP
# Option "NvAGP" "2" # use AGPGART
# Option "NvAGP" "3" # use AGPGART then NvAGP (default)
I could probably go with NvAGP 3, which should fallback to nvidia's AGP driver, but whatever...
I guess you should just make sure you have the AGP config the way it should be, either AGPGART enabled in the kernel and NvAGP 2, or AGP disabled in the kernel and NvAGP 1.
-(~) cat /proc/driver/nvidia/agp/status
AGP Rate: 4x
Fast Writes: Disabled
You should also know I had a real hard time getting nvidia's AGP driver to work. I think a driver upgrade fixed the problem, you may want to just go with AGP enabled in the kernel.