Right, but rivafb is the only driver under "framebuffer console" that has anything to do with nVidia, so a lot of people choose it as their console driver. Then they have problems with X.
If /proc/fb is empty, then are you even using a high-res console? Mine, for example, is 160x64 characters -- if yours is just 80x25, then it is indeed just normal text mode, and ignore what I've been saying.
Which distro is this, btw?