Yes, I do use the FB, but I use VesaFB, not RivaFB. If you think you have improved things by disabling the FB I'm going to be saddened. RivaFB has always been full of problems, but VesaFB worked pretty nicely.
Yes, I'm using the NVidia module and driver. I backrev'ed to 5328 last night after locking up 3 times: once while trying to play Quake3 and twice in X. But 5328 is no better, I woke up this morning to a locked X.
I'm using NV AGP (NvAGP = 1). I build AGPGART as a module now just in case, but found better stablity on NV AGP.
I really don't want to disable the FB.. I'm developing for the FB, and already pissed that RivaFB sucks ass.