It would probably be very helpful if you posted some information on your kernel. Like you, I had a lot of trouble getting AGP working when I initially tried the Linux drivers. In my case, the problem was that I was using a stock kernel and didn't have MTRR (Memory Type Range Register) support enabled. Once I enabled this option and rebuilt my kernel, everything worked great.
If you can post your kernel .config, someone may be able to point out the problem. Even if you're running a stock kernel, other users experienced with the distribution you are running may be familiar with the problem and able to help.
For reference, AGP support will make the most difference in performance when you are having to stream data from RAM to the video card. In extreme cases, this might be the case with textures, but scene geometry is more often the culprit (because it be kept in video RAM like textures can, because it changes from frame to frame). You may find that a game runs fine with just you in a level, but badly once other people are around and firing.