Re: [270.18] Overclocking is _still_ broken for 8xxx (probably 9xxx too) class GPUs
I took some time this morning to track down a G80-based card and was able to reproduce the problem.
The driver rejects the increased "core" clock frequency because the frequency of a related domain doesn't meet a ratio requirement in a path that sanity-checks incoming clock frequencies. This is due to a bug in one of the associated control calls. If the "core" clock frequency is increased in small increments, this bug can be worked around to a certain degree, as you discovered.
I tested a fix locally and will try to get it into an upcoming 28x.xx driver release. They say Better late than never, but I imagine that's bitter-sweet in this case. At the risk of sounding insincere, I do apologize for the terrible turn-around time.