Well i HAVE to speak to this point
First of all, if there is NOTHING running on a regular Xserver, there really should be not much memory being used, BUT if your using nvidia's drivers, than you will realize that it still uses a ton of resources. Nvidia's drivers aren't written to be extermely clean, as the xfree86 drivers are, Nvidia's drivers are written to be extremely fast.
- r0gu3