Reverting XF86Config wouldn't help if the old one loaded glx as well, and the illegal instruction was inside the libglx.so file. Which was why I was thinking that was the problem.
How about you get rid of ALL versions that you have (ESPECIALLY 4180, because 4180 isn't even for your processor), and try 3123 from scratch -- the .tar.gz 3123. Make sure you run X before installing the drivers, and try again immediately after installing them and modifying XF86Config. Then, if you get the same signal 4 error, revert XF86Config and uninstall the drivers with a make uninstall from the directory you did the make install from, and try X again. If it still doesn't work, then, well, you're seeing something very, very strange.
If it does work, then the problem may be that the files got corrupted. Possibly. Before decompressing, run md5sum on each tarball, and make sure the checksum matches what's posted on the driver download page, to try to prevent that.
Are you positive your memory, memory->CPU bus, L2 cache, and L1 i-cache are all flawless? If instructions are getting changed in one of those places, then that would do it. You might want to try memtest86 if you suspect one of those.