I had that happen when upgrading to the new drivers. My /etc/modules.conf still had the NVdriver aliased to char-major-195. The new drivers 4191 use nvidia now and not NVdriver. I think this happens due to the aliases file.
Edit /etc/modutils/aliases and /etc/modules.conf and change NVdriver to nvidia.
I cannot remember the correct debian way to do this, but it involves running the following after editing the aliases file:
In order to ensure that NVdriver is not loaded run the following (the real reason for your error I think is that NVdriver is loaded and running which prevents nvidia from getting it):
Make sure X is not running before doing the rmmod, but I assume it is not since you cannot install nvidia drivers. Then run the following:
That should hopefully do it.