Do you have SBA or fast writes on?
What chipset and processor are you using?
It's kernel 2.4.19, so this shouldn't be the Athlon XP/MP with AGP bug, because the feature that caused that bug has been disabled in that kernel...
What mode AGP (1x, 2x, 4x)?
Does using a lighter desktop help? Like fluxbox, enlightenment (although even e is a bit heavy), or even (gasp!) twm? KDE is a bit heavy on the resource usage, and that could conceivably have something to do with it.