Re: VGA to Component Video cable
The component output does work for 480p, 720p, and 1080i (and I think for 1080p as well). You don't define any modelines, however. You just set the TVStandard to HD1080i, and then set the resolution to "1920x1080" or the like. It's covered in the tv-out section of the README. One difference is that you can't switch modes between, say, 720p and 1080i output while running X. Nvidia said that this was a known bug, but it's never been fixed and it may even be a limitation of the tv-out. You're pretty much stuck using the proper resolution of the tv standard you use as well, so if overscan was something you compensated for with a modeline, that won't work.
I have no idea on the DVI connection, I was forced to get the component out bracket because I have a component-only tv. Have you tried connecting the board to a DVI LCD monitor to make sure that the fault isn't with the board itself rather than the convertor or tv?