I had the same problems. It seems to be a problem with some card BIOSes. Here's some details from the nvidia linux readme:
Q: X takes a long time to start (possibly several minutes). What can
A: Most of the startx delay problems we have found are caused by incorrect
data in video BIOSes about what display devices are possibly connected
or what i2c port should be used for detection. You can work around
these problems with the XF86Config option "IgnoreDisplayDevices"
(please see the description in (app-d) APPENDIX D: XF86CONFIG OPTIONS).