Benchmarking article at Tom's

[Smashed]

http://www17.tomshardware.com/column...624/index.html

Pretty good article giving some background on the recent benchmarking fiascos. Hasn't anyone else linked to it yet?

Coupla thoughts...

First off, I think all the furor about application detection misses the point. If a piece of code isn't well written, why needlessly cripple the hardware? Assuming the scene is rendered as intended, I see nothing wrong with shuffling operations, compressing multiple operations into fewer steps, or performing occlusion culling at an earlier stage. Perhaps the answer is allowing the application developers a chance to approve or reject these "optimizations"?

Obviously, though, occlusion culling should not be performed so early that it happens before the application is even launched. Nvidia's use of a clip plane in 3DMark 2003 was disgusting and borderline fraudulent. People would've been screaming bloody murder if they had done the same thing in a Splinter Cell or UT timedemo.

If it's such a terrible benchmark, Nvidia could've taken the high road and used their application detection to display a disclaimer warning that they do not support it. The way they put in a blatant cheat and started badmouthing 3DMark after they'd been caught really pissed me off. I don't expect them to admit to any wrongdoing or to apologize (any more than ATi did after the quack thing). I still think they got far less flak for it than they deserved.

Another interesting but not terribly surprising tidbit from the article is that ATi has been the main source of most of the embarassing revelations about Nvidia. ATi has been seen as taking the high road lately, but judging from some of their marketing material and how they're playing the online press, it seems they have no objections to fighting dirty themselves.

Weinand made a good point that sites should disclose their sources if the source has a significant stake in the matter. An Nvidia employee may need the protection of anonymity, but ATi certainly doesn't. For that reason, I didn't care for his poor attempts at humor, using coy phrases like "a 3d chipset company that does not wish to be named".

[digitalwanderer]

I dunno, one thing I didn't like about the article was that it was sort of back-handedly unbalanced.

It presented a pretty satisfactory overview of some facts, but it just implied things sort of weird to me. Like the way that ATi was "happily" digging up secrets on nVidia and dishing 'em out to fansites, while nVidia was "holding back" for some reason from divulging the terrible black secrets of ATi. (Yeah. Right. They're holding back damaging information on ATi out of a sense of fair play...what's wrong with that concept? )

[Sazar]

that was a nice read...

digi... don't forget what thomas wrote about the quake/quack issue when it first broke on [H] and who sent the info...

overall that was a better read than [H]'s article mainly because it REFUSED to condemn the benchmarks themselves completely..

definitely am along with many aspects of the text... the ihv's need to remember it is OUR money that has taken them to where they are

the least they can do is provide us with accurate... if inflated numbers w/o resorting to blatant hackery...

[rokzy]

the Tiger Woods screenshots are a great example of "the way it's meant to be played"

[Darth Rancid]

Hmm.. I have a distinct memory of Tom mentioning that nVidia pointed out "Quack" to him back in the days...

While Tom and his crew might be extremely competent, I feel they change their story so often they just can't be trusted...

[ChrisW]

And why shouldn't ATI inform these websites? I mean, by nVidia artificially inflating their scores, they are misleading the consumer by convincing them their products are faster than ATI's. Of course, this goes both ways. If nVidia knows of any deception by ATI, they should report this too. I want all cheats exposed by all graphics card companies.

[Sazar]

the article itself tries to take the POV of the parties concerned.. and does a decent job... its a nice little editorial considering the implications and what we have all known for a while

once sites like toms start putting out more information like this... rather than the [H] treatment... perhaps the IHV's will HAVE to act due to the massive readerships of these sites...

its only positive news...

Haven't read through it all yet...but so far my BS radar is going off already I'll edit this post when I finish the article.

edit: Ok, I finished the article. I think it was well written, although I think some of "facts" about the whole 3dmark03 debacle were not represented properly.

Specifically, http://www17.tomshardware.com/column...eating-06.html there is no mention that while Futuremark calls nvidias cheats optimizations now, they still say that the optimizations are NOT valid for the benchmark.

Also, http://www17.tomshardware.com/column...eating-07.html fails to mention that it is a FACT that nvidia lowered quality in the shaders, and this was proven by 3rd parties like B3D. ATI had nothing to do with exposing the famed 3dmark03 cheats that were defeated by the 3dmark03 330 patch. They did tip off website(s) to the 3dmurk03 AF issue. Quite frankly I think that's good. Better that we know about cheats than stay in the dark.

I thought the few ending pages were insightful enough.

Note: clipping planes(used in 3dmark03 and maybe other places) have absolutely nothing to do with occulusion culling. Tom's got this right too )

[Sazar]

Originally posted by StealthHawk
Haven't read through it all yet...but so far my BS radar is going off already I'll edit this post when I finish the article.

edit: Ok, I finished the article. I think it was well written, although I think some of "facts" about the whole 3dmark03 debacle were not represented properly.

Specifically, http://www17.tomshardware.com/column...eating-06.html there is no mention that while Futuremark calls nvidias cheats optimizations now, they still say that the optimizations are NOT valid for the benchmark.

Also, http://www17.tomshardware.com/column...eating-07.html fails to mention that it is a FACT that nvidia lowered quality in the shaders, and this was proven by 3rd parties like B3D. ATI had nothing to do with exposing the famed 3dmark03 cheats that were defeated by the 3dmark03 330 patch. They did tip off website(s) to the 3dmurk03 AF issue. Quite frankly I think that's good. Better that we know about cheats than stay in the dark.

I thought the few ending pages were insightful enough.

Note: clipping planes(used in 3dmark03 and maybe other places) have absolutely nothing to do with occulusion culling. Tom's got this right too )

there were some interesting fubar points

lol... that is correct...

there are elements of the editorial that are a bit off... but on the whole the article is a darn sight better than the [H] article on the same topic...

[Onde Pik]

*Checks article author*


Hmm... don't wanna waste my time on this.

[darkmiasma]

Originally posted by rokzy
the Tiger Woods screenshots are a great example of "the way it's meant to be played"

WTF are you talking about ... the ATI card is the one that is messed up in the Tiger Woods screenshots ...

or is that another shot at nVIDIA ... do you think they somehow convinced EA to make a broken shader path for ATI ...

[DivotMaker]

Originally posted by darkmiasma
WTF are you talking about ... the ATI card is the one that is messed up in the Tiger Woods screenshots ...

or is that another shot at nVIDIA ... do you think they somehow convinced EA to make a broken shader path for ATI ...

Conspiracy theories...

I can tell you that ATI was informed about this issue back before the game ever shipped and to my knowledge, EA's developer received no support from ATI to rectify the issue. I have also mentioned this issue before on the Rage3D boards in the driver section and nothing has yet emerged as far as a fix.

I can also say that it is asinine to suggest that EA would let any IHV (marketing agreement or no) convince them to "break" a part of ANY of their games just to "get back at nVidia's competitor"....how ridiculous.