nvidia.o is not an XFree86 driver!
It's a Linux kernel module, and should be in the /lib/modules/$(uname -r)/kernel/drivers/video directory. DO NOT MOVE IT.
When you say Driver "nvidia" in your /etc/X11/XF86Config(-4 if it exists) file, you are telling X to load the /usr/X11R6/lib/modules/drivers/nvidia_drv.o file. Not an nvidia.o, anywhere.
The reason you're getting an error seems to be that you've copied the kernel module into the XFree86 drivers directory, and for some reason, XFree86 is trying to load it instead of the proper file (the nvidia_drv.o file that comes from the NVIDIA_GLX package).
All you have to do is install the proper NVIDIA_kernel, then install NVIDIA_GLX, then edit your /etc/X11/XF86Config(-4 if it exists) configuration file, then restart X.
You don't really need to exit X if you don't have an older version installed.