This may help you as I just tracked down this problem and it may be related to your problem:
When starting X after installing the drivers the screen is distorted/scrambled with seemingly random characters and you cannot do anything to fix this.
Custom compiled kernel on Debian 3.0:
Kernel 2.4.20 from kernel.org
Geforce4MX 440, AGP, NV17
After many trials and errors by recompiling the kernel with different options I discovered that the problem was caused by enabling the following options:
Processor type and features -- >[*] Local APIC support on uniprocessors[*] IO-APIC support on uniprocessors
By not using theses options in my compile, the server works fine. So the problem is related to the APIC support and can be eliminated by compiling without these options.