Re: Low resolution with Nvidia drivers
Yes, setting a custom modeline as you have done should allow this to work.
I suspect that your monitor's EDID wasn't getting read for some reason. You can learn more about what exactly was happening by setting 'Option "ModeDebug" "True"' in the "Screen" section, restarting, and looking through the X log. With the ModeDebug option, you'll get verbose logging of the X server's mode validation.