I don't know your system, so I can't really comment on specifics. I have the following:
2 x amd 2600+ cpu
geforce4 ti 4600
2.4.20 custom kernel from kernel.org
I installed redhat, compiled/installed 2.4.20 kernel, built/installed both rpms, and I am having no probs.
post more specs, and maybe somebody can help...?