Except keep in mind that I don't see a high CPU utilization while starting X, EVERYTHING on the computer stops for several seconds. The screen stops updating, my real-time priority video captures stop working.
There may be high CPU utilization, but programs like top and such (even renice'd to the highest priorities) are never given a chance to update. Basically it's like the computer hangs for a little while (or at least that's the user's experience)
Does this sound like what anyone else is experiencing?
Oh, and Wolfman, I've got 256MB of RAM and I've got the same problem. I'm pretty sure it's unrelated to memory issues seeing as how I don't hear my hard drive doing any swapping while loading X a 2nd time.