I've seen this behaviour. Are you assessing size using 'top'? top was reporting my X server to be using 256MB of memory on a box with 128MB RAM + 128MB swap. The fact that the box worked quite normally suggested that the numbers weren't quite right somehow.
It's not clear to me that this is actual, used, physical memory. I wonder if it might be a shared-memory segment, or something.