Errrrr.... I've got the same setup (Gentoo, 2.4.20-gentoo-r6) with Highmem (1 gig of Ram) and it builds and loads fine. What options do you have in your /etc/make.conf? My CFLAGS are set to:
CFLAGS="-fstack-protector -O3 -mcpu=pentium4 -ffast-math -funroll-loops -fomit-frame-pointer -fforce-addr -falign-functions=4 -pipe -mmmx -msse -msse2 -fexpensive-optimizations -mfpmath=sse"
and it seems to work fine. Is your /usr/src/linux symlinked to the right directory? You could try the r6 kernel and see if that helps also (along with make clean make dep).