I had this problem when I tried to update my 2.4.22 slackware kernel with 5328 driver. No matter how I configured my kernel, I always had
(WW) NVIDIA(0): Failed to verify AGP usage
in the logs, and whilst the X mostly worked, I had to startx twice (1st one always crashed), and the 3D acceleration lagged like hell. I tried building the driver against 2.6.1 kernel, but configuration failed.
I solved it by stepping back to 4496 and 2.4.22 kernel, which works fine for now. X started and UT and Quake are running fine :P