Run top and press "M" to sort by memory to help diagnose the leaking app. Beyond that its arcane arts reserved for developers. Anyhow looking into it you will find that the mmaped buffers in X with 3D acceleration seriously screw your memory readings.
To bad about the pauses in ut must be anoying. Perhaps you could try changing agp drivers from nvagp to agpgart but Im just guessing here. It would be strange for linux to swap out an app that is in use, I know the vm sucks but not that bad. Good luck and have fun Chalnoth.