First, the main culprit is probably not X. It's probably the kernel caching filesystem data so that disk accesses are faster (and so that it doesn't take ~10 seconds to load up a directory with 8 files in it, like it does in Windows from time to time
To see the used/free counts without cached FS data included, you can take a look at the output of free
. In the +/- buffers/cache line, in the "free" and "used" columns, will be the amount of physical memory in use by all processes.
Now, as for X, it's not actually using 280M of memory. That's what it says in the VSZ column, but VSZ is virtual memory in use, not physical memory in use. In fact, X is using 280M of memory, but >90% of that is swap.
The RSS column (resident set size) is a better approximation of the amount of physical memory in use. As you can see, X only has ~23M of its memory locked into physical RAM. And this is the normal behavior of X, at least on my machines too.
It is normal for Linux to use all the memory it can get its hands on. This is why the OS is much less tolerant of memory errors than certain other ones.
Oh, the cached data quickly gets discarded, too, if real processes need more memory. So don't worry about that.