This COULD be an instability problem (of the whole system).
I've just gone through having an instable system, and I've run into a lot of weird behaviour, including (once) that X didn't get further than the spotted background.
If you are using AGP, then you could try if things improve when you turn it off with Option "NvAGP" "0". This might be enough, because the AGP architecture is one of the most instable busses ever designed; so, it might be that your system is instable, but not instable enough to not function without AGP. Note that it is also possible that turning off AGP helps because you ran into an agp driver bug - but in that case things should be more reproducible, less eratic. For example, you always have the exact same behaviour when starting X. Or you always get a black screen when running THAT application. If a system runs for a while and then crashes at seemingly random times - and/or when you see other errors, like like dma_intr error messages in your kernel log etc, then it's more likely an instability problem then a driver problem.
So, I'd say - start looking for randomness and/or reproducablity