This looks pretty nasty...
I happen to have the same graphics card (not VTO, though), although on top of a different chipset (VIA KT266). It always worked just fine for me.
Have you tried VIA forums? They have one dedicated to Linux here
. Also, make sure you check /var/log/messages as well, some probls might only appear there.
When you first start X (before computer goes crazy), what does 'cat /proc/driver/nvidia/agp/status' say? What about other files there?