I'm having the exact same problem.
Here are my system specs:
slackware 9.1 (experienced on both kernel 2.4.22 and 2.6.2)
using both 5.3.x drivers
Different card, different dist. different kernels...any ideas out there? I've used all variations of the AGP settings too to no avail.