I finally managed to install the latest 5328 drivers smoothly and and with all problems fixed.
For those with similar performance bottlenecks, the problem was a AGP-related bug with the pkg installer from the official nvidia download site. The fix is to go to http://www.sh.nu/download/nvidia/linux-2.4/
and use the installer from there.
I went from 0.3fps in tuxracer to around 1000fps!