Extremetech Just Updated their Cheating article with a *Correction* affecting ATi.

[Hellbinder]

Well
http://www.extremetech.com/article2/0,3973,1105259,00.asp

UPDATED ATI TEST RESULTS
In our original story down below, we presented extensive test data that showed how both ATI and nVidia were losing some performance in the newest 3.30 version of 3DMark03, which has implemented countermeasures to defeat app detection by 3D GPU drivers. Due to a variation in our test methodology, the drop seen in our initial tests for ATI appears to be have been unduly affected by the location where render state settings are made (in the application versus in the driver).
Our original test results misstated ATI's performance drop with 3DMark03 version 3.30, and we'll provide correct, updated results that set ATI in a much better light, with far less performance hit moving to version 3.30. However, ATI still detected 3DMark03 within their driver, and substituted a "better optimized" shader, and that is still benchmark cheating in our book, even if the overall performance boost was modest.

In other words, the results we initially reported were more pronounced than they should have been for ATI– holding all else equal. Based on our retesting, we still saw that ATI takes a small hit on Game Test 4 in version 3.30, which has to do with a driver "optimization" that ATI has already publicly stated will be removed from its next Catalyst driver.


We'll explain how these issues arose, and present the updated test results for ATI.


Wonder where this will lead. Heh,.. Whats it going to be a .05% *optomization* that everyone still calls a Cheat? The origional *cheat* was only 1.9% as it is. :rolleyes:

[deejaya]

However, ATI still detected 3DMark03 within their driver, and substituted a "better optimized" shader, and that is still benchmark cheating in our book, even if the overall performance boost was modest.

I think that is the important part. To be honest I couldn't give two craps how much any company "cheats", it shouldn't happen. Measuring cheating is a bit lame.

[Hellbinder]

I think that is the important part. To be honest I couldn't give two craps how much any company "cheats", it shouldn't happen. Measuring cheating is a bit lame.

Hmm.. Thats where we part company. First ATi did not Substiture the Shader, they took the exact same shader and slightly reordered the instrction execution. Which imo is a 100% Valid *optimization* and it would be the Same for Nvidia if thats all they did.

3dmark03 is supposed to be the *gamers* benchmark. Designed to represent the performance of Dx9 *Games*. This is exactly the kind of optimization s that take place is all games. It is exactly what I want to see ATi doing for me as a user. the difference between what ATi did and Nvidia did is a large Vaste Gulf. Nvidia lowered the FP percision to below DX9 Spec, Completely Replaced the shaders with custom ones of their own, and inserted Clip planes.

Surely at least some of you can see the difference.

[creedamd]

Originally posted by Hellbinder

Surely at least some of you can see the difference.

The ones that don't have the thick green glaze can, Hellbinder. Just so other people will respect you a little more, a couple questions.

If the evidence was on the other foot and it was Ati that was under the gun, would stick to your opinions? Would you blast Ati as well?

[Hellbinder]

If the evidence was on the other foot and it was Ati that was under the gun, would stick to your opinions? Would you blast Ati as well?

Thats a Good Question. Honestly i would hope so. I have defended Nvidia a time or Two in the past on various things.

Part of the equation is that my Understanding of Hardware and Software has Grown a great deal in the last 24 months. It is perhaps True that 2 years ago I would have Hammered Nvidia for doing shader reordering but not today.

You should also check out my post in the Nvidia Forum where I address the details of what really is the Difference in DX9 capabilities and why we are where we are today with all the Fuss.

[deejaya]

Originally posted by Hellbinder
Hmm.. Thats where we part company. First ATi did not Substiture the Shader, they took the exact same shader and slightly reordered the instrction execution. Which imo is a 100% Valid *optimization* and it would be the Same for Nvidia if thats all they did.

3dmark03 is supposed to be the *gamers* benchmark. Designed to represent the performance of Dx9 *Games*. This is exactly the kind of optimization s that take place is all games. It is exactly what I want to see ATi doing for me as a user. the difference between what ATi did and Nvidia did is a large Vaste Gulf. Nvidia lowered the FP percision to below DX9 Spec, Completely Replaced the shaders with custom ones of their own, and inserted Clip planes.

Surely at least some of you can see the difference.

In no way am I condoning what nVidia did, it's obvious to anyone that the combination of things that nVidia done far outweighed what ATi did. But nothing changes the fact that they both targeted and changed that benchmark at driver level, to artificially increase performance, which is cheating, no matter what way you look at it.

If optimizations like these were used in games, it can't be called cheating, as it benefits the entire game, from start to finish, it's an optimization. But in a benchmark the only legitimate optimizations should be across the board optimizations that don't do anything application specific to 3dmark, and apply to everything the card does, in all situations.

Yes, there is a difference, but the difference between them lies on the wrong side of the line here - they were both in the wrong.

[Hellbinder]

You are missing the point that the exact same water and Sky Shaders Could easily be used in any of todays games. Thus the exact same Changes to these two specific areas would be expected in any game or application.

[creedamd]

Originally posted by Hellbinder
Thats a Good Question. Honestly i would hope so. I have defended Nvidia a time or Two in the past on various things.

Part of the equation is that my Understanding of Hardware and Software has Grown a great deal in the last 24 months. It is perhaps True that 2 years ago I would have Hammered Nvidia for doing shader reordering but not today.

You should also check out my post in the Nvidia Forum where I address the details of what really is the Difference in DX9 capabilities and why we are where we are today with all the Fuss.


I learn a lot from your posts. I respect your "Understanding of Hardware" I read from your posts that it isn't because of the name "Nvidia" that you have these quarrels, that it is for legitimate reasons.

[Hellbinder]

i keep trying to tell people that i am really a PowerVR fanboy without a product. But no one believes me.

You wait Till PowerVR series 5 Hits the street in Q3/4 this year. There are going to be some Shocked people out there. :D

[deejaya]

Originally posted by Hellbinder
You are missing the point that the exact same water and Sky Shaders Could easily be used in any of todays games. Thus the exact same Changes to these two specific areas would be expected in any game or application.

I know they could be, but the fact is they weren't. They were specifically re-ordered for 3dMark, and 3dMark only. The bench was detected, and altered by the ATi drivers. The Futuremark audit clearly states:

If the drivers do their job correctly, 3DMark produces the precisely same rendering on all hardware.

Re-ordering shaders definitely means that 3dMark is not doing precisely the same rendering job on all hardware.

[GlowStick]

Hmm, your view is pritty good accept i disagree that ATi's is useing optmizations.

What is happening is when ATi's driver sees the specific shader from 3dmark03 it intercepts that shader, and replaces it with one IN the drivers. eg.

Lets say 3dmark03 sends this code to the compiler

Instruction A
Instruction B

the drivers intercept and replace it with the tweaked (this code was written by a human)

Instruction B
Instruction A

That is exactly what Nvidia is doing, but they are useing a diffrent shader made to run even faster.

To NOT CHEAT the Real Time Compiler would have to do this

any program sends any shader routine

Instruction A
Instruction B

the COMPILER analizes the code, then runs this (this 'code' was not made by a human, but an algorhytim that works for all shaders)

Instruciton B
Instruction A

How do we know that is not going on?
With the new patch that makes 3dmark03 undetectable (for now) the preformance gain is lost.



Also, what ATi and Nvidia did is exactly like makeing a Custom Path for their video card. But they went behind FutureMarks back and did not tell them about it, untill they got caught.

If you fully support ATi's as not cheating at all not cheeting 100%, then useing the NV30 path on Doom 3 is 100% a fair benchmark between ATi and Nvidia cards.

But useing the NV30 path is infact, a little bit diffrent, the path was created by the game programmer, not changed behind the back of the game programmer.

[digitalwanderer]

Originally posted by Hellbinder
Surely at least some of you can see the difference.

Yeah, I can and I agree with you...but I ain't gonna defend ANY optimizations for benchmarks as I really don't think they are right/fair/honorable. (That is my OPINION, not hard fact!)

I agree with you, but I'm also glad they're removing the optimizations from future driver sets just to remove even the air of impropriety from their drivers...in the current atmosphere it's a very understandable and I think commendable attitude. :)

Originally posted by Hellbinder
i keep trying to tell people that i am really a PowerVR fanboy without a product. But no one believes me.

You wait Till PowerVR series 5 Hits the street in Q3/4 this year. There are going to be some Shocked people out there. :D

I'll be one of the first to pick one up if'n you recomend HB, I always loves me new hardware to play with and a whole new company would be OODLES of fun! :D

[Hellbinder]

I can fully understand why you guys disagree with the *optimization*. I cant really argue with your Viewpoint of the issue as it is as much a Valid as mine is. If you want to look at this from a completely Puritan View then yes Both Companies are Cheating.

[R.Carter]

If ATI drivers were always able to optimized arbitrary shaders for any application would optimizing shaders still be considered cheating?

[digitalwanderer]

Originally posted by Hellbinder
If you want to look at this from a completely Puritan View then yes Both Companies are Cheating.

I can understand your more technical and deeper view of it too Hellbinder, but if we start pushing the argument for ATi's optimization it just gives ammo to nVidia's anti-cheat arguments.

I feel it's just too technical a distinction for most people (by "most people" I am NOT refering to many on these boards! ;) ) to understand and it definately would take me longer to explain to a newb than it would take for his eyes to glaze over at all the geek-speak.

So ya, I'm looking at it as a bit of a black-n-white issue when I know it ain't. Sorry, I just feel it's the best way for now or I would be spending me whole life arguing the finer points of distinction between cheating & optimizing online and this activity already chews up enough of me free time. ;)

[GlowStick]

Originally posted by Hellbinder
I can fully understand why you guys disagree with the *optimization*. I cant really argue with your Viewpoint of the issue as it is as much a Valid as mine is. If you want to look at this from a completely Puritan View then yes Both Companies are Cheating.

Ok, i can see you point also, and i do like the fact that even with *it* turned off, your not loseing an arm and a leg in your score.

But if anyone cares, here is how i would make 3dmark03 a 'good' benchmark.

First, No one gets copys of the developr's version of 3dmark03.

Second, when a new video card comes out, or there is a need for some video cards to be compared, REVIEW sites get a Special version of 3dmark03.

The review sites do not talk to ati or nvidia about it. And this 3dmark03 is diffrent, it is not the same as everon downloads and buys.

It uses diffrent shaders, made for a one time use only, not optmized for card A nor card B or C etc. You just have to trust Futuremark on that one. EG, the water in GT4 could be changed to fire, just to make it diffrent.

Next, the camera path is changed, the same Geomortry and Textures can be used.

Finally, the program is alterd so it 'cannot' be detected, just like updateing Starcraft or something, to get rid of the 'hacks'.

This will mean though, you can only compare the scores on cards running the SAME 'version' of the bench. But that is just fine for reviewsites, they just need to note that.

What this allows Futuremark to do, is ATi and Nvidia do not get a chance to optmize drivers for JUST 3dmark03.

Thats just my 'vision' of 3dmark03 though, could be flawed!

[PreservedSwine]

Originally posted by deejaya
. But nothing changes the fact that they both targeted and changed that benchmark at driver level, to artificially increase performance, which is cheating, no matter what way you look at it.

If optimizations like these were used in games, it can't be called cheating, as it benefits the entire game, from start to finish, it's an optimization.

That's not true. The types of optimizations ATI used in 3dm03 are considered optimizations by virtually every one within the industry. All ATI did was re-order some shader instructions to be run in parralell. The output is IDENTICAL, just quicker. This is exactly the type of optimization we would see in a game....

[gRoUnD]

Hellbinder it is good to see you here. I hope we all get around somehow, just like old Beach Boys. Stick to facts plz :)

All of you

[DivotMaker]

Originally posted by Hellbinder
Thats a Good Question. Honestly i would hope so.

If you are as self-righteous about "right and wrong" as you want us to believe here, then your answer should have been

Absofreakinglutely!,

instead of

Honestly i would hope so

I know my answer would have been Absofreakinglutely!.....

[deejaya]

Originally posted by PreservedSwine
The output is IDENTICAL, just quicker. This is exactly the type of optimization we would see in a game....

Well the visual output may be identical, but the benchmark output is not identical. I quoted it earlier, I'll quote it again, from the Futuremark audit:

3DMark on the other hand is designed for this very purpose. If the drivers do their job correctly, 3DMark produces the precisely same rendering on all hardware. Some hardware with a more limited feature set actually do more work in order to achieve the same rendering, but the resulting rendering is the same, the workload is comparable, and the performance result is justified as a world wide standard.

Re-ordering shader instructions specifically, for 3dMark alone, using application detection, means the workload is different for the ATi card. It may render it visually the same, but it's not doing the same on the ATi card as it is on the Matrox (I won't say nVidia for obvious reasons ;) ) It's not doing it how Futuremark intended.

And Tim Sweeney was commenting on his code being "optimized", and John Carmack said something like it was "morally grungy, but not deeply evil". That's not exactly everyone in the industry.

The only people who could really make the call here would be Futuremark. It's their benchmark, and they lay down the rules. I am only stating my opinions, and I can see both side of the lines. I just think that 3dMark03 should benefit purely from optimizations that work across the board, and aren't specifically made for the benchmark, and the benchmark alone.

[Sazar]

Originally posted by BigBerthaEA
If you are as self-righteous about "right and wrong" as you want us to believe here, then your answer should have been

Absofreakinglutely!,

instead of

Honestly i would hope so

I know my answer would have been Absofreakinglutely!.....

yes lets get into personal attacks... its the only way we can get into discussions here init ?

:rolleyes:

[Hellbinder]

It just unbelievable what goes on around here. Dont you guys realize that its BECAUSE of some of the exapples of posts made in this thread that I get upset and out of hand sometimes???

People in this Forum are a Self-Fullfilling Prophecy why is it that the mods cant see it? I generally dont go out of my way to make endless personal attaks on people, Or make pointless posts in the middle of thoughtfull discussions.

Look at some of the Junk tossed into this thread. Some people are so Blind they cant see the Forest For the Trees.

[DivotMaker]

Originally posted by Sazar
yes lets get into personal attacks... its the only way we can get into discussions here init ?

:rolleyes:

Personal attack? OMFG.... Had enough of the double standards for one evening......

[Hellbinder]

If you are as self-righteous about "right and wrong" as you want us to believe here, then your answer should have been

Absofreakinglutely!,

instead of

Honestly i would hope so

I know my answer would have been Absofreakinglutely!.....

Not Suprisingly you only partially Quoted me. I also said

Part of the equation is that my Understanding of Hardware and Software has Grown a great deal in the last 24 months. It is perhaps True that 2 years ago I would have Hammered Nvidia for doing shader reordering but not today.

Which should tell you that My entire post from the begining was one of Personal Reflection to make sure i was on the up and up with my comments.

Again you illustrate a propensity to make personal attacks instead of addressing the Information. If you dont agree with the Specific points that I make and even Insist are RIGHT. Then Make a case agaist it and prove me wrong. That is how a Forum works.

To just get pissed off becuase you dont like the information is Trueley as you put it earlier.. Without Class.

[Hellbinder]

Personal attack? OMFG.... Had enough of the double standards for one evening......

Again, I attack peoples POINTS not their Person Generally. With teh specific exeptions of Behemoth, Nv40, and BKSwansky. There are some very specific reasons I ahve gon after them in the past. Generally though i STILL usually debate them based on their posts. Not even Every one of their posts.