Benchmarking article at Tom's

[Smashed]

http://www17.tomshardware.com/column/20030624/index.html

Pretty good article giving some background on the recent benchmarking fiascos. Hasn't anyone else linked to it yet?

Coupla thoughts...

First off, I think all the furor about application detection misses the point. If a piece of code isn't well written, why needlessly cripple the hardware? Assuming the scene is rendered as intended, I see nothing wrong with shuffling operations, compressing multiple operations into fewer steps, or performing occlusion culling at an earlier stage. Perhaps the answer is allowing the application developers a chance to approve or reject these "optimizations"?

Obviously, though, occlusion culling should not be performed so early that it happens before the application is even launched. Nvidia's use of a clip plane in 3DMark 2003 was disgusting and borderline fraudulent. People would've been screaming bloody murder if they had done the same thing in a Splinter Cell or UT timedemo.

If it's such a terrible benchmark, Nvidia could've taken the high road and used their application detection to display a disclaimer warning that they do not support it. The way they put in a blatant cheat and started badmouthing 3DMark after they'd been caught really pissed me off. I don't expect them to admit to any wrongdoing or to apologize (any more than ATi did after the quack thing). I still think they got far less flak for it than they deserved.

Another interesting but not terribly surprising tidbit from the article is that ATi has been the main source of most of the embarassing revelations about Nvidia. ATi has been seen as taking the high road lately, but judging from some of their marketing material and how they're playing the online press, it seems they have no objections to fighting dirty themselves.

Weinand made a good point that sites should disclose their sources if the source has a significant stake in the matter. An Nvidia employee may need the protection of anonymity, but ATi certainly doesn't. For that reason, I didn't care for his poor attempts at humor, using coy phrases like "a 3d chipset company that does not wish to be named".

[digitalwanderer]

I dunno, one thing I didn't like about the article was that it was sort of back-handedly unbalanced.

It presented a pretty satisfactory overview of some facts, but it just implied things sort of weird to me. Like the way that ATi was "happily" digging up secrets on nVidia and dishing 'em out to fansites, while nVidia was "holding back" for some reason from divulging the terrible black secrets of ATi. (Yeah. Right. They're holding back damaging information on ATi out of a sense of fair play...what's wrong with that concept? :eek2: )

[Sazar]

that was a nice read... :)

digi... don't forget what thomas wrote about the quake/quack issue when it first broke on [H] and who sent the info... :)

overall that was a better read than [H]'s article mainly because it REFUSED to condemn the benchmarks themselves completely..

definitely am along with many aspects of the text... the ihv's need to remember it is OUR money that has taken them to where they are :)

the least they can do is provide us with accurate... if inflated numbers w/o resorting to blatant hackery...

[rokzy]

the Tiger Woods screenshots are a great example of "the way it's meant to be played" :rolleyes:

[Darth Rancid]

Hmm.. I have a distinct memory of Tom mentioning that nVidia pointed out "Quack" to him back in the days...

While Tom and his crew might be extremely competent, I feel they change their story so often they just can't be trusted...

[ChrisW]

And why shouldn't ATI inform these websites? I mean, by nVidia artificially inflating their scores, they are misleading the consumer by convincing them their products are faster than ATI's. Of course, this goes both ways. If nVidia knows of any deception by ATI, they should report this too. I want all cheats exposed by all graphics card companies.

[Sazar]

the article itself tries to take the POV of the parties concerned.. and does a decent job... its a nice little editorial considering the implications and what we have all known for a while :)

once sites like toms start putting out more information like this... rather than the [H] treatment... perhaps the IHV's will HAVE to act due to the massive readerships of these sites...

its only positive news...

[StealthHawk]

Haven't read through it all yet...but so far my BS radar is going off already :p I'll edit this post when I finish the article.

edit: Ok, I finished the article. I think it was well written, although I think some of "facts" about the whole 3dmark03 debacle were not represented properly.

Specifically, http://www17.tomshardware.com/column/20030624/nv_cheating-06.html there is no mention that while Futuremark calls nvidias cheats optimizations now, they still say that the optimizations are NOT valid for the benchmark.

Also, http://www17.tomshardware.com/column/20030624/nv_cheating-07.html fails to mention that it is a FACT that nvidia lowered quality in the shaders, and this was proven by 3rd parties like B3D. ATI had nothing to do with exposing the famed 3dmark03 cheats that were defeated by the 3dmark03 330 patch. They did tip off website(s) to the 3dmurk03 AF issue. Quite frankly I think that's good. Better that we know about cheats than stay in the dark.

I thought the few ending pages were insightful enough.

Note: clipping planes(used in 3dmark03 and maybe other places) have absolutely nothing to do with occulusion culling. Tom's got this right too :) )

[Sazar]

Originally posted by StealthHawk
Haven't read through it all yet...but so far my BS radar is going off already :p I'll edit this post when I finish the article.

edit: Ok, I finished the article. I think it was well written, although I think some of "facts" about the whole 3dmark03 debacle were not represented properly.

Specifically, http://www17.tomshardware.com/column/20030624/nv_cheating-06.html there is no mention that while Futuremark calls nvidias cheats optimizations now, they still say that the optimizations are NOT valid for the benchmark.

Also, http://www17.tomshardware.com/column/20030624/nv_cheating-07.html fails to mention that it is a FACT that nvidia lowered quality in the shaders, and this was proven by 3rd parties like B3D. ATI had nothing to do with exposing the famed 3dmark03 cheats that were defeated by the 3dmark03 330 patch. They did tip off website(s) to the 3dmurk03 AF issue. Quite frankly I think that's good. Better that we know about cheats than stay in the dark.

I thought the few ending pages were insightful enough.

Note: clipping planes(used in 3dmark03 and maybe other places) have absolutely nothing to do with occulusion culling. Tom's got this right too :) )

there were some interesting fubar points :D

lol... that is correct...

there are elements of the editorial that are a bit off... but on the whole the article is a darn sight better than the [H] article on the same topic...

[Onde Pik]

*Checks article author*


Hmm... don't wanna waste my time on this.

[darkmiasma]

Originally posted by rokzy
the Tiger Woods screenshots are a great example of "the way it's meant to be played" :rolleyes:

WTF are you talking about ... the ATI card is the one that is messed up in the Tiger Woods screenshots ...

or is that another shot at nVIDIA ... do you think they somehow convinced EA to make a broken shader path for ATI ... :rolleyes:

[DivotMaker]

Originally posted by darkmiasma
WTF are you talking about ... the ATI card is the one that is messed up in the Tiger Woods screenshots ...

or is that another shot at nVIDIA ... do you think they somehow convinced EA to make a broken shader path for ATI ... :rolleyes:

Conspiracy theories...:rolleyes:

I can tell you that ATI was informed about this issue back before the game ever shipped and to my knowledge, EA's developer received no support from ATI to rectify the issue. I have also mentioned this issue before on the Rage3D boards in the driver section and nothing has yet emerged as far as a fix.

I can also say that it is asinine to suggest that EA would let any IHV (marketing agreement or no) convince them to "break" a part of ANY of their games just to "get back at nVidia's competitor"....how ridiculous.

[digitalwanderer]

Originally posted by BigBerthaEA
I can also say that it is asinine to suggest that EA would let any IHV (marketing agreement or no) convince them to "break" a part of ANY of their games just to "get back at nVidia's competitor"....how ridiculous.
Not meaning to flame you here BB, but why is it so asinine to suggest that nVidia might be using under-handed tactics/pressuring developers for favoratism? I'm not meaning to impune EA here, but nVidia has given lots of people lots of reasons to think exactly such things and I do not believe it is "asinine" to assume them capable of such. :(

And on the screenshots, is the nVidia sky supposed to look like it's just a solid blue-fill or is that just something weird with the way they captured that particular screenshot?

[DivotMaker]

Originally posted by digitalwanderer
Not meaning to flame you here BB, but why is it so asinine to suggest that nVidia might be using under-handed tactics/pressuring developers for favoratism? I'm not meaning to impune EA here, but nVidia has given lots of people lots of reasons to think exactly such things and I do not believe it is "asinine" to assume them capable of such. :(

And on the screenshots, is the nVidia sky supposed to look like it's just a solid blue-fill or is that just something weird with the way they captured that particular screenshot?

I don't believe I was commenting on nVidia. I believe I said EA would not allow nVidia to pressure them into doing such things assuming that nVidia actually did such things. I am also speaking STRICTLY about the Tiger Woods development team. I know the TW Producers at EA and the development team at Headgate Studios extremely well and those people would not stand for tactics like that...

I believe the gamma of the two screenshots to be very different, but that is EXACTLY the differences I see between the 9800 Pro and the Ti4600.

[Sazar]

Originally posted by BigBerthaEA
I don't believe I was commenting on nVidia. I believe I said EA would not allow nVidia to pressure them into doing such things assuming that nVidia actually did such things. I am also speaking STRICTLY about the Tiger Woods development team. I know the TW Producers at EA and the development team at Headgate Studios extremely well and those people would not stand for tactics like that...

I believe the gamma of the two screenshots to be very different, but that is EXACTLY the differences I see between the 9800 Pro and the Ti4600.

besides the gamma and the some corruption on the 9800pro screenie... is there nothing else you notice on the 5900u/9800pro comparison on the screenie ? :)

[digitalwanderer]

Originally posted by BigBerthaEA
I believe the gamma of the two screenshots to be very different, but that is EXACTLY the differences I see between the 9800 Pro and the Ti4600.
Do you mean that when you try running Tiger Woods on a 9800 & GF4 that the only real visual difference is the gamma levels or the weird solid filled sky? (Again, I'm just honestly confused here and am looking for clarification...no fighting on this end. ;) )

[DivotMaker]

Originally posted by digitalwanderer
Do you mean that when you try running Tiger Woods on a 9800 & GF4 that the only real visual difference is the gamma levels or the weird solid filled sky? (Again, I'm just honestly confused here and am looking for clarification...no fighting on this end. ;) )

No. The 9800 screen looks exactly like the output I get on my 9800 Pro.

The GF FX screen looks exactly like the output I get on my Ti4600...

[digitalwanderer]

Originally posted by BigBerthaEA
The GF FX screen looks exactly like the output I get on my Ti4600...
Ok, thanks. My question is does the FX have just a solid filled sky on your game too? Because aside from the water reflections that was the thing that sort of jumped off the screenshots and hit me square between the eyes like a hammer....it was a bit noticeable.

Is it like that or was it just the screenshots had different weather effects going?

[Sazar]

the water hazard on the OTHER side of the hole is not rendered on the gf FX screenie... either that or it is infested with algae :D and shadows under trees are kind of off...

the water also looks different to the 9800pro's water (disregard the texture corruption)..

and yes... the sky is a solid color rather than having variance...

[digitalwanderer]

Originally posted by Sazar
the water hazard on the OTHER side of the hole is not rendered on the gf FX screenie... either that or it is infested with algae :D and shadows under trees are kind of off...
Doh! I can't believe I missed that! Yup, you're correct...definately something different there too.

[Sazar]

Originally posted by digitalwanderer
Doh! I can't believe I missed that! Yup, you're correct...definately something different there too.

first thing that strikes people are the artifacts... but the rest of the image appears to be rendered better on the 9800pro.. or at least with more detail...

even the water looks better... albeit massively artifacted (the better part == non-artifact ridden)

however... I am still trying to figure out wtf lars was trying to imply with this comparison ?

this is similar to the flat water issue some nvidia cards have with splinter cell... I'd wager a genuine bug or other such issue thats just waiting for resolution...

[StealthHawk]

Originally posted by Sazar
there were some interesting fubar points :D

lol... that is correct...

there are elements of the editorial that are a bit off... but on the whole the article is a darn sight better than the [H] article on the same topic...

I just recently skimmed the [H] editorial. Quite frankly I don't care much for what Kyle says about these issues. he has shown that he is either ignorant, biased, or clueless on the whole matter. Or some devious combination of the above. I mean, you either want to do a technical review with apples to apples comparisons or you want to do a "gaming review" as I have stated before. There really is no point in trying to go half way on each.

[DivotMaker]

Originally posted by Sazar
the water hazard on the OTHER side of the hole is not rendered on the gf FX screenie... either that or it is infested with algae :D and shadows under trees are kind of off...

the water also looks different to the 9800pro's water (disregard the texture corruption)..

and yes... the sky is a solid color rather than having variance...

Just noticed the missing water hazard in the FX shot. Not sure why that is.

The skydome looks to be the one that is used for "Sunny" conditions. Too bad you can't see the lens flare effects in either of those shots. All the others have volumetric clouds that move with the wind direction...it is really cool to play in Stormy conditions and watch the rain pelt you and leave little puddles on the ground, especially the green. Now if we could just render a "rooster tail" for the golf ball when it is putted in the rain, then we might be ON to something....

Believe it or not, the Radeon 9800 water might look better in a still shot, but the GF renders the animated water a bit more realistically. Hopefully ATI might offer some assistance this year in working some of these rendering issues....who knows?

[Behemoth]

Originally posted by Sazar
first thing that strikes people are the artifacts... but the rest of the image appears to be rendered better on the 9800pro.. or at least with more detail...

even the water looks better... albeit massively artifacted (the better part == non-artifact ridden)

however... I am still trying to figure out wtf lars was trying to imply with this comparison ?

this is similar to the flat water issue some nvidia cards have with splinter cell... I'd wager a genuine bug or other such issue thats just waiting for resolution...
so have ati fixed the game yet? they always seem to keep customers wait for some fixes for months. no offense.

[Sazar]

Originally posted by BigBerthaEA
Just noticed the missing water hazard in the FX shot. Not sure why that is.

The skydome looks to be the one that is used for "Sunny" conditions. Too bad you can't see the lens flare effects in either of those shots. All the others have volumetric clouds that move with the wind direction...it is really cool to play in Stormy conditions and watch the rain pelt you and leave little puddles on the ground, especially the green. Now if we could just render a "rooster tail" for the golf ball when it is putted in the rain, then we might be ON to something....

Believe it or not, the Radeon 9800 water might look better in a still shot, but the GF renders the animated water a bit more realistically. Hopefully ATI might offer some assistance this year in working some of these rendering issues....who knows?

commenting basically on the images in the tom's review and a recent b3d review which had flat water issues... :)

personally I have had no problems ingame so far... and iq has been great...

the only FX gpu I have personally used has been the 5800 (for a little bit) and a 5200 gpu that my friend has... :) therefore I will reserve judgement on these till I actually spend a decent amount of time with em :)