I've had this experience for a LONG time. It have happened in Windows 2000, Windows XP and now Gentoo Linux.
I have been talking to my vendor, MSI, and they said that the Nvidia reference drivers weren't working well with their cards. But their drivers are too old to be used for serious gaming.
Anyways, my system FREEZES too, for 15-20 seconds when starting X. And it is slow when Alt-switching to a console. Also, in Windows XP, when in the "advanced display properities" clicking the "Geforce4" tab, my system freezes too. But my WinAmp buffer is usually large enough to compensate for it. This only lasts for a few seconds though, but it is basically the same thing. Also starting XP have been very slow in the past, just like X. With the latest drivers 41.09, I believe, it is a bit better.
The nvidia-driver issue is really the one thing that really annoys me. Any chance of improving the responisveness when switching to/from X, or when starting X? It really is a bitch. Also, I wonder if this may have something to do with the motherboard? I will consider startup responsivness/init time as an important factor when choosing my next GFX card, sometime in february. It is crap to have a system that boots in 15 seconds, with a GFX card that uses 20 sec. to initialize.
Mine is a VIA KM133, AtlonXP 1800, 512MB 133Mhz SDRAM, GeForce4 MX440.