Did some more tests. The vgahw module loads whether there is an entry in the XF86config file. So - like the professor kindly suggested, I can’t bypass vgahw’s loading. Anyway, I tried turning off dbe, type1, freetype – even glx.
Now by altering the startup timings, I saw that the slowdown may not have anything to do with vgahw itself anyway. Slowdown had already begun before loading vgahw, and given that the system is in complete molasses, my timings are unreliable during the slowdown.
So I tried something else. I tried running X in gdb (with no symbols...). The slowdown starts immediately after xf86VidMem() is called. More precisely after an xf86ioctl() call.
In my XFree log, it looks like the start of the slowdown corresponds to
(--) NVIDIA(0): MMIO registers at 0xF4000000
I downloaded the XFree86 source tree and will do some more tests later.