I built a new computer a few days ago and I had no luck with the latest drivers either. I managed to get into X only once, and doing so required me to throttle AGP down to 4x, and even then it froze after only a couple of minutes. It's been working great since I downgraded to the 3123 drivers, however according to the /proc/driver/nvidia entries, the card now only supports AGP 1x and 2x, whereas with the 4191 drivers it reported back 4x and 8x. (The card is 8x capable.) You'd think that an NVidia card running on an NVidia chipset would work well, so I'm hoping that they can fix this soon.
Athlon XP 2100+
Abit NF7 (Nforce2 chipset)
512MB PC2700 DDR
Asus V9280/TD Geforce4 Ti4200 8x
Gentoo Linux w/ 2.4.20 kernel