Did you have to do any special things with .so files (shared libraries) to get the rpmbuild program to work? If so, it might be that using the wrong libraries is causing the segfaults. But I have a feeling that that isn't the problem...
Is this machine overclocked? Is the memory good (have you tried compiling a full kernel ever? -- don't bother running it, just configure and make bzImage and watch for segfaults there, as that's a pretty decent tester); has it been tested recently, like with memtest86 or similar? How about temperatures inside the case, specifically, on the processor? Have you ever tried to get anything like lm_sensors working, to monitor the temperature?
Which compiler are you using? (cc -v will tell you that) Is there a chance that it's slightly buggy?