Do not be frightened of big values in 'top' or 'ps'. These programs shows memory segment sizes associated with given process, neither the physical memory usage nor "common" virtual memory (physical + swap). Just nvidia driver uses device memory mapping (mmap("/dev/nvidia0",..) into process memory space) and uncommitted SysV IPC shared memory. Most of this memory are "pure virtual", i.e. does not associated with physical pages, but part of them are associated with AGP memory.
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2349 root 9 0 304M 15M 4824 S 0,0 6,2 8721m X
try to run cat /proc/2349/status (where 2349 is my PID of X, replace it by your own):
VmSize: 329020 kB
VmLck: 4294922036 kB
VmRSS: 15956 kB
VmData: 44248 kB
VmStk: 464 kB
VmExe: 1468 kB
VmLib: 4976 kB
We can see that X uses only 15MB of physical memory (RSS - Resident Set Size) and 44248 +464+1468+4976 (i.e. ~50MB) of "common" virtual memory (so ~35 MB in swap), where 5,5MB (VmExe and VmLib) can be shared (possibly) with another processes (another
copies of X for VmExe and another processes using the same shared libraries for VmLib).