Yes, KDE slows you down. Especially if it was:
1) Compiled with an old binutils (that doesn't combine relocation information), or loaded with an old ld.so (one that doesn't cache the last symbol resolution)
AND, if it was:
2) Not objprelinked. But objprelink works only on i386-class processors (Intel/AMD), and you can sometimes see a lot more segfaults than normal, so distributions are highly unlikely to enable this.
If it was prelinked, or if it was compiled with a newer binutils and you're using a newer ld.so, then you'll see less of a slowdown, but the fact is that KDE is pretty big, and therefore fairly slow. Gnome isn't much better.
twm doesn't have all the "desktop" functionality, it's a bare-minimum window manager. Actually, it's barely even that. It comes with X, so as long as you have X installed, you can use it if you want. To use it, create a .xinitrc file (the dot at the beginning is important) in your home directory, containing the following lines:
exec xterm -name login
When you exit out of the xterm named login, your X session will end. You will also have to place the xterm windows yourself when X starts up (there are no defaults, unless you pass a -geometry argument to xterm -- click the mouse to put them down somewhere), but if you're only going to be doing this to run a benchmark, I don't think it'd be a huge deal. When you want KDE back, mv the .xinitrc file to some other name, and restart X.
There are two X config files because XFree86 version 3 used an incompatible syntax to what XFree86 version 4 uses. So the XFree developers figured they'd make it look for its own syntax file first, then the old one if its version wasn't found. It's so you can have and use X 3 and X 4 on the same system.