That's not an issue with the refresh rate. It's a known problem with the 4191 drivers.
nVidia, for reasons unknown, decided to do away with accelerating the RENDER X extension and replace it with something proprietary that apparently doesn't work very well. Making your Gnome background a solid color (rather than a pixmap, aka picture) will help, if you use Gnome. Otherwise, trying 3123 is about all I can think of suggesting, unless I forgot something...
If you do a forum search for something like "4191 slowdown", you should find enough threads that one of them should have a few things to try to speed it up.