I had a similar problem with my brandnew Geforce4 Ti4400 128MB. What happens is that sometimes the card will perform extremely poor, right from booting up! Today I experimented with Twinview and video-out a bit, without much success I might add. However, when I restored everything back I found that performance of my system was very bad e.g. even text mode was sluggish i.e. GRUB boot menu was veeery slow. I "fixed" this by connecting my main monitor to the second monitor out (DVI port with DVI->VGA converter cable). When booting the system up with the monitor connected to the second head everything was back to normal! I reconnect the monitor to the VGA port and it worked full speed.
The solution might sound very strange, but that is how I fixed it. So I think it might have something to do with the card misdetecting the connected output device and using some sort of reduced speed mode or something. Perhaps some NVIDIA gurus can shed some more light on this matter?
So chances are good this problem has nothing to do with GNOME/KDE.