I have the same problem as you describe. When I startx as root it takes ages for it to start up. When I use a display manager like xdm or kdm X won't start at all, I'll just get the nvidia logo for a moment, then see a the xdm background for about half a second and the screen turns to black again for about 2 minutes, this goes on indefinitely (well at least for a couple of hours). Setting the IgnoreDisplayDevices option to "DFP, TV" fixes that issue but now I can't use nvtv anymore to use tvout (which makes sense I guess). It can't find any modes.
Didn't have this problem with the older drivers (although I cant remember exactly at wich version this problem was introduced).
Is there a solution?
Asus A7V266-E (via kt-266a)
Athlon XP 1700+
(But the same thing happens on both debian & gentoo)