X is not KDE. In order to run the vast majority of games, you NEED to have X running.
However, yes, KDE will slow the game down a bit. Perhaps not enough to notice, but those ten tons of KDE processes will slow stuff down eventually.
If you start X up with just the client program (using something like startx /path/to/ut2003), then you'll be running with no window manager or desktop environment at all, just the UT2k3 binary. Once UT2k3 exits, so will the X server.
You can even do that on a separate display if you need to -- but if you've got KDE running on the first, it'll still take its share of CPU time.
And most of these comments also apply to Gnome, too. twm, though, is pretty lightweight, as are the *box window managers (blackbox, fluxbox, etc.).