If you really want to downgrade follow the following steps:
1) do "updatedb" in a console
2) do "locate libGL"
3) remove all libGL files you got from locate (except libGLU and perhaps libGLw)
4) do "locate libglx"
5) again remove all libglx files
6) remove /usr/X11R6/lib/modules/drivers/nvidia_drv.o
7) remove /lib/modules/(YOUR_KERNEL_VERSION)/kernel/drivers/video/NVdriver
After this your install should be fresh again.