More info .. Solved maybe?
I have been up for over 3 days now by disabling framebuffer. It was VesaFB that I was using btw.
I think it was the combination of agpgart, framebuffer and nvidia. Anyway, I've got agpgart and nvidia and it seems more stable.
technikolor: have you tried NvAGP "1" in your XF86Config-4 file, and disabled agpgart in the kernel? That fix worked for me on my other system with the same problems. I can't do it with amd64 though, as nvidia's agp driver doesn't work for that platform.