Re: Nvidia Driver 275.28
I'm seeing what I believe are similar problems. I run 64-bit Debian Sid (unstable). The package maintainers switched from Nvidia driver version 280.13 to 275.28 to match the upgraded xserver-xorg-core ABI bump from 1.10 to 1.11. After I upgraded the Xorg and Nvidia driver packages, my Gnome desktop became horrendously slow. Even just switching window focus acquired a very noticeable delay. When I looked at top, the CPU didn't seems to be particularly active, and I didn't see any outlandish or rapidly-growing memory usage (no where near hitting swap).
I've since downgraded the relevant Nvidia driver packages, using the IgnoreABI option to allow it to run against the newer xserver-xorg-core. For a few minutes, the responsiveness seemed almost back to normal, but it degraded pretty quickly. I'm going to try downgrading the xorg package to see if I can return to normal conditions.