On a whim I tried doing startx through ssh, just to see what would happen, and I got this...
(==) Log file: "/var/log/Xorg.0.log", Time: Sun Oct 21 19:43:17 2007
(==) Using config file: "/etc/X11/xorg.conf"
(II) Module "ramdac" already built-in
(EE) NVIDIA(3): Unable to find available Display Devices for screen 3.
X in realloc(): error: junk pointer, too high to make sense
X connection to :0.0 broken (explicit kill or server shutdown).
The bit about screen 3 makes sense because nvidia-xconfig seems to have made a bogus Screen. The realloc() part is more interesting.
I tried removing the bogus screen from the config and still got the junk pointer error.