many thanks, you saved the day. As soon as I disabled GPU scaling, everything worked. I'll forward this to linux-bugs(a)nvidia.com, so that they can fix this unbelievable pitfall...
Originally Posted by poltsy
Check if GPU scaling is forced enabled in nvidia-settings as it will force default refresh rate and will cause all your settings to have no effect on the actual output even though all the tools will say you're getting what you asked for.
Note that settings you change in nvidia-settings will only be applied once you run it so you have to run it every time X restarts. If you run it as root it will apply settings from /root and if you run it as user it'll get them from the user $home so if you're not always running it as the same user the behaviour might seem a little indeterministic. I'm not really sure if running it as root has any purpose to begin with.
If that doesn't actually change anything you could start the X server with -logverbose 6 to get a better idea what exactly it is about the modeline that makes it fail the validation (and adjust any limits accordingly).