The standard Matrox driver does use AGP, but it looks like there's a framebuffer driver as well--is that what you meant?
I tried turning off AGP, but left DRI on, and the framebuffer code as modules, and still no luck. This morning I started configuring with AGP and DRI off, and the fb code included (not as a module), but I didn't have time before work. We'll see about that tonight.
Also, where is the agpgart patch you mention? I assumed it would be in the source tarball, but all I see is code for audio and network.
Finally, I'm curious--how likely is AGP support for non-nVidia cards?