Ok, I've been running these drivers over the last week without a reboot and things are really starting to grind down. Even just opening a new terminal window seems to lag the system so the point where xorg is completely unresponsive to the mouse for a few seconds.
Doesn't appear to be any noticeable increase in memory usage but during these slowdowns xorg cpu usage can get upto 90% WCPU usage in top. I'm not running any fancy compositing settings, just plain xfce4.
Also nvidia doesn't seem to appear in vmstat -i any more so I can't see if the interrupts are going crazy, just that top says 0.9% interrupt.
I've attached a bug report. Just going to reboot and see if that helps.
Rebooted and now its all back to full speed again, very smooth.