My post was a bit premature... I've found a solution to this problem!
After reading all the information provided by nVidia I added the following line to the device section of XF86Config:
Option "IgnoreDisplayDevices" "DFP, TV"
(since I use a CRT display) and X now starts after only a couple of seconds