I'm actually having the same problem after upgrading from XFree 4.2 to XFree 4.3 and using the latest nVidia drivers. I think it has more to do with XFree86 4.3 than with nVidia. Try emerging XFree 4.2 and see if you still have the problem.
I'll keep you guys posted if I can figure out what's wrong with 4.3.
"Floating in a dream-like state, I am the emporer of a parallel universe." --Arch Enemy
My xorg.conf file