My thoughts on "Optimization"

[digitalwanderer]

Originally posted by jimmyjames123
They are not consistent in their methods for "normalization". To exclude non-WHQL drivers for their results browser, while at the same time allowing overclocked gpu's and cpu's, is not very consistent IMHO.

I still just don't get that one. It's like saying that a race isn't fair because all the cars use different engines & transmissions, while the "optimizations" are flat-out changing the track one car is driving on.

And this is pure speculation. We don't know the details so all we can do is speculate. However, considering that they don't have authorized access, if NVIDIA did use the developers version that would be considered "cheating" as well correct? Or would that be simply level the playing field?

I don't know, but if they used it to include clip planes in to artificially inflate the score and invalidate the test than I'd consider that "cheating".

digitalwanderer, I read your post at ATI Rage3d fan site (you have 6000 posts there, wow!).

Yup and about 4500 at DriverHeaven and another 3600 or so over at RageUnderground even though I'm permanently banned there (even though it doesn't SHOW I'm permanently banned from there...Andre is such a cheeseball! ). I got bored on Friday and added some of the sites I hang at most and figure I got about 15-16K posts or so...what's your point?

"nvidia is flat-out freaking busted cheating in their new FX drivers!". For some reason, I gather that you take more joy out of this issue than it really warrants.

Now that is entirely a matter of opinion and I figure I can get me jollies over whatever I wish. I don't deny I'm enjoying nVidia's getting busted, they've been full of hubris for way too long and it's damned refreshing to see 'em suffer for it now.

Remember we are only talking about a benchmark that many people feel is not representative of real-world gaming performance, and we are talking about an issue that doesn't actually affect image quality in what we can normally see. The FX Detonator 44.03 drivers have shown significant improvements in image quality and performance for a wide variety of gaming benchmarks for the FX cards.

Again, the relative values of 3dm2k3 are a subjective call...but the fact that nVidia is short-cutting the intent of the test is not.

You can argue if it's right or wrong of them to short-cut the intent, but methinks the fact that they are is pretty well established.

nVidia has been flat-out lying to the consumer lately and YES that honks me off! VERY much so, I find it reprehensible.

They have a great product, there is no reason for them to lie about it!

[jimmyjames123]

I haven't seen this "wide variety of gaming benchmark boosts".

Do a google search. There are at least one or two out there.

Is it only Nvidia that can't submit non-whql scores? Because of Nvidia EVERYONE has to use whql drivers to submit a score. How is this unfair to Nvidia? Does Nvidia not make whql drivers just like everyone else?

Actually, it is because of Futuremark that EVERYONE has to use WHQL drivers. And the current Detonator FX 44.03 is WHQL certified for FX cards. So what's the big deal?

Are only SOME cards allowed to submit overclocked scores? Last I checked EVERYONE was allowed to submit overclocked scores. How is this unfair to Nvidia?

Fair or not fair is not the point. The issue is normalization and "accuracy" of a supposedly neutral benchmark. The whole idea behind WHQL was normalization presumably. If you are going to allow overclocked scores, that helps undercut this goal. The cards that overclock better will have the natural advantage. It is a inconsistently on FM's part, that's all.

You said they talked to ATI privately for a month about this before going public with it. You show me where they said this. You won't be able to show me this.

You are completely full of it. I read it at [H]OCP, from Kyle's very own mouth! Why don't you email Kyle about it? I'm not going to search through a ridiculously large thread just to satisfy your tastes.

[John Reynolds]

Originally posted by jimmyjames123
If that is truly the case, then it is the consumers who are losing because we already know that 3dmark performance is not necessarily representative of performance in real-world games, and we already know that both NVIDIA and ATI can "optimize" their drivers for improved performance in 3dmark.

Chant the word "optimize" three times and maybe it'll come true, ok? :rollseyes:

That certainly seems like an overstatement. "Optimization" such as this can apparently only be done with a fixed camera view where we know exactly what will be rendered. So this is an issue that should not be evident in typical games. The Detonator FX 44.03 drivers have been tested by several professional reviewers, and their image quality and performance seem to be up almost across the board for the FX cards. Regular FX users have verified this too if you read some forum comments.

The clip planes "cheat" requires a fixed view angle to work. So do timedemos. Still with me? That isn't too complicated of a "cheat" to understand, is it? The fact that this "cheat" can be used in a game's timedemo more easily since they don't run at normal frame rates like 3DM does should be cause for alarm. Until we know for certain that Nvidia didn't also "cheat", say, UT2003's timedemos we should play it safe and rule out any scores based on these drivers. "cheat"

[John Reynolds]

Originally posted by jimmyjames123
The visuals count in the sense that the goal is to maximize combination of "high" image quality and "high" frame rates. One without the other will leave a noticeable hole. The final score is important to a certain degree, but if people do their homework they will know better than to judge based on one synthetic benchmarking number.

No, the visuals are a result of calls made to the hardware's drivers. So long as those calls aren't being tampered with, an equivalent amount of work will be done. The graphics rendered will not be identical between different products because those products aren't identical. However, API standards exact a certain level of specifications for how these things work, a standard that is defined by, in this case, MS (with input from the IHVs). An optimization is when a way is find to more efficiently manage the process by which the drivers communicate between the chip metal and the calls being made, not by manually finding ways to ignore the calls.

The problem is that there seems to be no concrete definition about which "path" to follow. Things are just not so simple unfortunately when talking about graphics performance.

Pure BS. You see the path, and regardless of its twists and turns, you start from point A, its start, to point B, its finish. Any deviations from that path is not an optimization, but a cheat. A true optimization would be the athlete getting his butt into better shape before the race is run.

The sad thing in my analogy is that the cheating athlete could very well have won the race, and might still in the future, but because his management has chosen to act unethically the illegal shortcut was taken.

[Ady]

Get over it already. I know that you're not going to change my mind no matter what is said. It's a cheat, it's plain and simple.

It's also rather obvious that..

1. Ati fans will think it's a cheat. Even if they don't really understand or read up on it.
2. Nvidia fans will think it's NOT a cheat. Even if they don't really understand or read up on it.
3. People that don't care either way will still judge this as a cheat.

[jimmyjames123]

I still just don't get that one. It's like saying that a race isn't fair because all the cars use different engines & transmissions, while the "optimizations" are flat-out changing the track one car is driving on.

It's not that difficult a concept really. Your analogy if flawed because what we actually see from 3dmark is not corrupted at all with the FX driver, so in that sense they really are "driving on the same track" (although maybe they are skipping a behind-the-scenes pit stop). If Futuremark excludes non-WHQL results from their database in an effort for more normalization, then it's really inconsistent to include overclocked results, especially since overclocked results can skew the data much more than WHQL certification.

I don't deny I'm enjoying nVidia's getting busted, they've been full of hubris for way too long and it's damned refreshing to see 'em suffer for it now.

I'd say there is a problem if you think it is "damned refershing" that any company that provides good products for the consumer should be suffering. Still, it's not very appropriate to bring that attitude to a NVIDIA forum IMHO.

nVidia has been flat-out lying to the consumer lately and YES that honks me off! VERY much so, I find it reprehensible.

I'd say that the Detonator FX 44.03 drivers should be very welcome by FX card owners. Almost without exception, they have increased image quality and performance across the board. This is certainly much more important to a gamer than what happens in a synthetic benchmark.

[Sazar]

Originally posted by jimmyjames123
Do a google search. There are at least one or two out there.

Actually, it is because of Futuremark that EVERYONE has to use WHQL drivers. And the current Detonator FX 44.03 is WHQL certified for FX cards. So what's the big deal?

Fair or not fair is not the point. The issue is normalization and "accuracy" of a supposedly neutral benchmark. The whole idea behind WHQL was normalization presumably. If you are going to allow overclocked scores, that helps undercut this goal. The cards that overclock better will have the natural advantage. It is a inconsistently on FM's part, that's all.

You are completely full of it. I read it at [H]OCP, from Kyle's very own mouth! Why don't you email Kyle about it? I'm not going to search through a ridiculously large thread just to satisfy your tastes.

let me get this straight... you are saying BECAUSE someone can overclock a card... it makes that cards score invalid ?

what is the logic behind this ?

there is a HUGE difference between a WHQL/non-WHQL driver submission and submission of results based on overclocking... and I fail to see ANY correlation between the two based on your arguments...

also... yes kyle did say he had discussions with ATi officials for 1 month before going public with the quake/quack information... and yet.. he went public word for word with what nvidia had given him incl usage of their program to an extent to identify the mipmap patterns to show what was going on...

btw... futuremark is not forcing video card makers to release WHQL drivers

isn't it m$ that certifies these drivers... hence is it not m$ pushing for a certain level of quality and compatability...

futuremarks demands for WHQL drivers is based on the fact that it then becomes harder for a set of drivers to be released solely for the purpose of benchmarking since it will have to WORK in everyday situations as well... the intentions are good and that is a plus...

[jjjayb]

Actually, it is because of Futuremark that EVERYONE has to use WHQL drivers. And the current Detonator FX 44.03 is WHQL certified for FX cards. So what's the big deal?

Exactly. What's the bid deal with having to use whql drivers? Except that the non-whql drives you want used lower precision below dx9 standards.

You are completely full of it. I read it at [H]OCP, from Kyle's very own mouth! Why don't you email Kyle about it? I'm not going to search through a ridiculously large thread just to satisfy your tastes.

Posted by Kyle on his Radeon 8500 review on Friday, October 19th, 2001 in his 8500 review:

We did not get the card till Tuesday morning of this week, the day before Matt Waters, a HardOCP writer, bought one to play with for himself, so we had a card to verify our results with.

This is also where he first discusses quack:

We all know that Quake3 is one of the most utilized 3D benchmarks in the world, there is simply no arguing this. We came into possession of a program that you will see more of here soon. It is an .exe file that when placed in the Quake3 directory renames every single instance of the word "quake" to "quack" as well as builds a new quake.exe file named quack.exe. When you run quack.exe, you will find that Quake3 runs exactly as it should.

So tell me, who is full of it? You can dig through the Hardocp forums all you want and you will never find what you claim to be there. It only exists in your head. The only one full of it here is you.

[jimmyjames123]

The clip planes "cheat" requires a fixed view angle to work. So do timedemos. Still with me? That isn't too complicated of a "cheat" to understand, is it? The fact that this "cheat" can be used in a game's timedemo more easily since they don't run at normal frame rates like 3DM does should be cause for alarm.

Well duh this isn't too complicated to understand. And there are people who feel that such timedemo's are not a great benchmarking tool because they can be susceptible to such "optimizations" (say it three times please).

Until we know for certain that Nvidia didn't also "cheat", say, UT2003's timedemos we should play it safe and rule out any scores based on these drivers.

We don't know what has happened in UT2003 timedemos, so we can't say one way or another. I think I read somewhere that timedemo's change every time you run them, is this true? This certainly doesn't invalidate all the other true gaming benchmark's though (like Doom 3, Quake 3, etc).

[Sazar]

Originally posted by jjjayb
Exactly. What's the bid deal with having to use whql drivers? Except that the non-whql drives you want used lower precision below dx9 standards.



Posted by Kyle on his Radeon 8500 review on Friday, October 19th, 2001 in his 8500 review:




This is also where he first discusses quack:



So tell me, who is full of it? You can dig through the Hardocp forums all you want and you will never find what you claim to be there. It only exists in your head. The only one full of it here is you.

if you go through hardOCP you will find kyle or frgmaster... whatever floats his boat... did actually come out and say he had been in discussion with ATi for about a month...

of course he has said this within the past week and there was no prior notification of any transactions between [H] and ati that I am aware of or have read online... which probably explains your disbelief @ this... but having read this myself I can assure you that kimmy is not making THIS up...

[jjjayb]

So tell me Jimmjames, do you disagree with websites using ut2k3 timedemo benchmarks? What does an ut2k3 flyby demo benchmark have to do with the actual game? When in the game do you "flyby" the scenes. The ut2k3 fly-by demos have NOTHING to do with actual game performance, yet everyone still uses them in their reviews. So far I haven't heard any complaints about that. Only compaints that 3dmark03 doesn't represent actual game performance.

[jimmyjames123]

let me get this straight... you are saying BECAUSE someone can overclock a card... it makes that cards score invalid ?

No I did NOT say the scores are invalid. I said it makes any attempts to "normalize" seem undercut.

there is a HUGE difference between a WHQL/non-WHQL driver submission and submission of results based on overclocking

It's really not that hard to understand. It's a question of normalization. FM only wants WHQL results in an attempt to normalize things. However, by allowing overclocked results for the online results browser, their standards become inconsistent. If you don't see that, there is no simple way that I will convince you otherwise.