The default NVIDIA .run install file supports your card and OS. Try http://download.nvidia.com/XFree86/L...-4496-pkg2.run
as it supports 2.4.20-8smp and Quadro4 550. Follow the normal directions in the README file.
If that doesn't work, use the same file and tell it you want to compile the module vs using the pre-compiled one.