I don't think it is a problem with agpgart. I've tried disabling it and it makes no difference. I 've also enabled log in the nvidia module and it seems that the module keeps running ( it keep logging stuff) even when my X is frozen. I think it is a bug in the X server because it happens with differen card . I've seen the problem on a Dell server at work with a Quadro FX500.
I also got the Nvidia card because it is fast and I thought well supported. Now I'm back to my original ATI card.