New 3DMark03 patch out - nVidia still cheating

[DMA]

Originally posted by AnteP
ROFL!

Hey, don't laugh. Thank me for the good tips instead. I promise you, the kids out there reading these pathetic biased reviews won't even see how bad it is. They'll only see ATI on top and BAM!! Job done.

Well, i'm off. I gotta edit my latest review and try to add some scores from the latest build of 3DMark-03.
Haven't used that bench for months but this is too good to leave out.


Go ATI!! :dance:

[TheTaz]

I agree with Eaun... some points that people are forgetting.

1) It's a Direct X 9 Benchmark. It's meant to show STANDARD coded Direct X, Not IHV specific paths. (This, of course, does not give an OpenGL picture... so to use it as "THE ONLY" Benchmark to go by is ridiculous... It also doesn't give a picture of any IHV specific paths that Devs may be using for actual games... and It was NEVER MEANT TO.)

2) Since it's a Benchmark, It deligates a SIMULATED workload. It's not intended to reflect "REAL GAME" performance. It's intended to show how much a piece of hardware can do under a FUTUREMARK'S heavy SIMULATED workload (Way heavier than games). A lot of benchmarks use 'simulated workloads'... that's why they are Benchmarks, not real applications!

3) In order for this "Benchmark" to accurately depict the capabilities of 'said hardware', NO SHORTCUTS can be allowed. (As Euan pointed out... if there are things that are purposely giving out more workload for SIMULATION purposes, making shortcuts that doesn't affect the Image is STILL a CHEAT, in a BENCHMARKING situation, because you are circumventing the 'assigned workload.').

That said... why are people upset if this isn't meant to reflect real game performance? Because, since this app is used as a PR tool (By both IHV's and OEM's), cheating (Or more nicely put... circumventing the workload) in it, is equivalent to "consumer fraud".


Let me try to put this in perspective:

My friend has two nForce 2 boards.

One is an older ASUS A7N8X Dual channel 333MHz FSB and an Athlon XP 2700+ Tbred.
The other is a newer ASUS A7N8X-X Single channel 400MHz FSB and an Athlon XP 3000+ Barton

Stock clocking, SiSoft Sandra shows the older System beating or near equal the newer system in MANY of the tests (CPU's are clocked near the same if you look it up... actually the 2700+ is clocked a *tiny* bit higher). It doesn't reflect the gaming performance. It only reflects the *general* hardware capabilities *within SiSoft's rules*. (Most games actually perform a little better on the newer motherboard, due to the extra cache and FSB speed).

Bottom line... if you *break the rules* of a benchmark... the benchmark is INVALID. Futuremark realizes this, and is attempting to protect the validity of THEIR *ruleset*.

Whether you believe FutureMark's application is a good tool to test with, or not, doesn't mean ANYTHING. What matters is, if it's a tool being used as PR to persuade consumers. And since IHV's and OEM's are using it as a PR tool to persuade consumers... IT MUST BE UNTAINTED in it's "ruleset".

/em steps off the soapbox

Regards,

Taz

[ChrisW]

The question is which is more reflective of future (DirectX 9) games? Before patching to 340 or after? Which shows a closer relative performance difference between the two cards (nVidia and ATI) with games like Tomb Raider (which is already optimized for nVidia cards)? Game after game seems to agree more with the 340 patched version than before.

EDIT: Has anyone checked their scores with the new 52.70 driver set?

[ChrisRay]

I'm more concerned about Game Test 2 and 3 than 4,


I've always read that the vertex skinning in Game Test 2 and 3 are unrealistic. But, How different is the r300's and Nv30 series vertex shader?

[The Baron]

You could say that they were using clip planes. Were they? I really don't know. It's possible, sure, but I don't think NVIDIA is that stupid since it's easy to catch (or at least was).

[dan2097]

You could say that they were using clip planes. Were they? I really don't know. It's possible, sure, but I don't think NVIDIA is that stupid since it's easy to catch (or at least was).

I was under the impression that according to dave from beyond3d the 44.67s has a clip plane in GT2 although that was all that came of that, no update/screenshots

[The Baron]

Originally posted by dan2097
I was under the impression that according to dave from beyond3d the 44.67s has a clip plane in GT2 although that was all that came of that, no update/screenshots
Hm, I thought GT2 just didn't clear the back buffer.

[StealthHawk]

Originally posted by The Baron
Hm, I thought GT2 just didn't clear the back buffer.

The bag of tricks is very deep.

Although over time there have been some nice legitimate gains(or undetectable illegitimate ones :p ) as you can see from my performance numbers(you know the thread).

[AnteP]

Originally posted by DMA
Hey, don't laugh. Thank me for the good tips instead. I promise you, the kids out there reading these pathetic biased reviews won't even see how bad it is. They'll only see ATI on top and BAM!! Job done.

Well, i'm off. I gotta edit my latest review and try to add some scores from the latest build of 3DMark-03.
Haven't used that bench for months but this is too good to leave out.


Go ATI!! :dance:

Why can't I laugh, I just love conspiracy theories.
Yours is especially funny. :afro2:

To be honest though I assume your accusations are guided towards me since I'm the only person right now that uses a combination of the games and settings you mention.

To clarify how the games are chosen here we go:

I choose games on a few criteria:
I look at reseller listings (ie top sales list)
I try to browse the internet to find what games seem to be popular at the moment
I try to find statistics on which online games are popular
I also try to find a good blend of different games (all the way from pre-TNL up to DX9 and both OGL and DX)
Sometimes I also chose games which I personally like very much (Mafia is the game I'm referring to here)

I also have some limitations when I do my selection:
I have to have access to a game to benchmark it (ie either the dev/publisher sent it to me or I go out and buy it)
if not then at least there must be a demo that's fully representative of the full version game (something that the UT2003 demo is not for an example)
There also must be some easy way to benchmark the game, if not a benchmark mode then at least some predectability so that I can use FRAPS

I also listen to ATI and nVIDIA since they often have recommendations on which games to test and not to test
usually I rather listen to which games they'd prefer me not to test and if their arguments stand valid I'll exclude the mention titles.
this hasn't happened yet that I can remember though

As for the settings I chose 4xAA/8xAF since I think, at 1280x1024, it gives a perfect blend of acceptabel performance and very good image quality.
I could of course go with a higher res but fact is that many people still have monitors that are limited to lower res or at least they cant run 16 by 12 in a high enough refreshrate to make it a viable options.

I personally belive that a high end 500 dollar video card should, in the winter of 2003, produce acceptable framerates in most games with these settings.

As for not testing more settings it's simply a matter of weighing in the options.
If I have more settings I'd either have to test less games or else I'd have to test less boards. Since there simply isn't enough time.
Also I personally think that unless performance is way way below par there's simply no reason to test with low image quality settings.
Just as we don't see any sites benchmarking with just bilinear filtering, or 16 bpp nowadays.
It's evolutions, the older stuff has to go sometime. I let it go a few months ago since I think we've finally reached a point where basically ALL videocard available except for the sub 75 USD perhaps can perform to some extent even with AA/AF.

There you have it. Feel free to throw more conspiracy theories spiced up with sarcasm. :)

Just to make you personally more content I actually e-mailed nVidias PR manager today and asked him what he though of my test suite of 12 games and the settings I use.

And lat but not least: no 3dmark is not making a return back into any review of mine except perhaps when a new architecture is presented.
I use games, since games is what you use the board for.
Synthetic benchmarks have their place in exploring new architextures to find out their strengths and weaknesses in the preliminary tests.

If you have any questions please let me know and I'll be happy to answer them. And of course if/wen I get the reply from nVidia Ill let you know what they think about the test suite/settings.

Cheers.

EDIT:
as for the TWIMTBP games I simply don't care, TWIMTBP is a marketing program, I won't let such activities interfere with my work as an editor.
To be honest I don't even keep track of which games are and are not part of that program. The ones I can mention of the top of my head are UT2003 and Tomb Raider. (As for tom raiders inclusion in the suite it's simple: it's the first DX9 game availabkle comercially and thus is of interest in my opinion)
As for Quake 3 I thought it would be nice to have a legacy game in there. Besides quite a few players still play Q3 and the mods for it. (Including my boss hehe) But to be honest I just stuck to it because it's probably THE easiest game to benchmark ever, it takes like ten seconds to benchmark, you get predictable results and it's just a one click process if you use Q3Bench)

So there ya have it.

[NickSpolec]

I don't understand how Nvidia feels it has the right to optimize 3DMark03 because they feel their "unoptimized" 3DMark03 score is not indicative of their actual performance in games.

Maybe someone should show Nvidia their performance numbers from most games, like Max Payne 2, XIII, or Tomb Raider: AOD.

[Uttar]

Originally posted by AnteP
Why can't I laugh, I just love conspiracy theories.
Yours is especially funny. :afro2:

Hehe.
But you know what's the most frightening?

That review's conclusion would be nearer to the truth than 75% "objective" reviews. No kidding.

Quake 3 would be considered "old" and the other games "new". The conclusion would thus have to be "NVIDIA is faster in old legacy software, but ATI clearly dominates new games with absolutely stunning graphical details and astonishing polygon counts!"

And considering how register usage is higher in more complex programs and it degrades performance... And that other things make more complex shading even slower compared to ATI, such as less potential usage of FP16...
That conclusion would be exagerated, but more technically accurate than the ones on Tom's, Anand or [H].

Ironic, isn't it? :angel:



Uttar

[cthellis]

Originally posted by ChrisW
The question is which is more reflective of future (DirectX 9) games? Before patching to 340 or after?

Funny thing is you can pretty much use both to tell. 340 will more closely resemble the starting point, and 330 will more closely resemble how it will look if it's popular enough to get enough driver attention from nVidia. ;)

[Sickness]

Originally posted by ChrisW
The question is which is more reflective of future (DirectX 9) games? Before patching to 340 or after? Which shows a closer relative performance difference between the two cards (nVidia and ATI) with games like Tomb Raider (which is already optimized for nVidia cards)? Game after game seems to agree more with the 340 patched version than before.

EDIT: Has anyone checked their scores with the new 52.70 driver set?

Yes the 340 patch is more reflective of future DX9 development. Nvidia will have to optimize for the NV3x architecture until they release the NV 40, of course they may not bother much at all after they rectify their design problems.

Apparently the 52.70 set has the same results.

[Hellbinder]

Originally posted by Joe DeFuria
Which means exactly nothing with respect to real games. 3DMark tests show that unless nVidia detects the application, and has hand tuned optimizations for a specific app, you're not getting squat.

If their "compiler" were so magical, then they wouldn't need to detect the application or use empirical data. The nVidia's compiler were genuine, the patch would not impact their scores.
This is exactly what i keep saying.

It is so obvious that this new "Optomizer" is nothing more than a simple PR tool to cover up that they are still doing the same things they were before.

The only difference is they have had an additional 6+ months to work on replacement shaders and application specific "optimizations" so that it is far less noticable. The Results of their Work are Definitly very GOOD no one can deny that. But lets at least be honest about whats going on. The only down fall is that they cant replace every shader or game code that does not suit them in every game that gets released. The ones that they do "fix" will go "unfixed" until the next release.

This is only a "bad thing" If you are a consumer or an Nvidia user that cares about it. All i really want people to see is that this is indeed what they are approving for themselves and not making decisions off of Deciefull PR and misrepresentations about what is really going on.

[Hellbinder]

Originally posted by Rogozhin
"An optimization must accelerate more than just a benchmark unless the application is just a benchmark."

this was the two sentences being brought together-it's a totally valid statement.

I've not built any systems with nvidia for over 4 months and my clients are happy and content.

It is rhetorical garbage even without the mould.

rogo
That Nvidia Statement is Complete both Complete Hypocracy and nonsense at the same time.

"An Optimization must accelerate more than just a benchmark"

It is plainly obvious what this means. There are numberous benchmarks out there and they were out there long before this statement was made. Therefore it is obviously talking know they are talking about BENCHMARKS. It is clear that Nvidia knows what they are saying here. DO NOT OPTOMIZE FOR A BENCHMARK ONLY. Good statement. Eeveryone applauded and agreed.

Now Nvidia has been cought "Cheating" again. And they come back with this..

"An optimization must accelerate more than just a benchmark unless the application is just a benchmark."

making themselves out to be complete Liars for the First statement. Where they had openly acknowledged that benchmarks should receive no special optiomizations directed only at them. Now they are "Claiming" they ment jsut the oposite and that its "ok" to Optomize for benchmarks.

I dont know what is sadder. They Nvidia employees would stoop so low as to say such a thing? or that there are many people out there who will not only accept this statement but agree with them.

[DaveBaumann]

I suggested that HB, not NVIDIA.

However, I think the real issue here is not the 3DMark performance, but whether the optimisation guidelines they have reiterated to the press on two different occasions now are actually real or not. Upon first looks it would appear that the 52.16 violates all three of their guidelines in 3DMark. Are they serious about these guidelines or are they just paying lip service to appease the press?

Thats you you should think about IMO.

[Sazar]

Originally posted by DaveBaumann
I suggested that HB, not NVIDIA.

However, I think the real issue here is not the 3DMark performance, but whether the optimisation guidelines they have reiterated to the press on two different occasions now are actually real or not. Upon first looks it would appear that the 52.16 violates all three of their guidelines in 3DMark. Are they serious about these guidelines or are they just paying lip service to appease the press?

Thats you you should think about IMO.

has FM come up with any formal agreement as to what will happen if there are continued violations or perceived violations Dave?

afaik there is no stated actions to be taken in such a case but i could be mistaken...

[ChrisRay]

Originally posted by Sazar
has FM come up with any formal agreement as to what will happen if there are continued violations or perceived violations Dave?

afaik there is no stated actions to be taken in such a case but i could be mistaken...


They just wont recognise the results on there orb, Nothing futuremark can really do about it. Cept Kick Nvidia out of their program. Which wasnt good for them last time. ( I mean Nvidia being out of the program, Not being kicked by futuremark)

It was just give Nvidia more PR Leverage to use against them and appear like a victom. And i have no Doubt Nvidia would so such a thing.

[digitalwanderer]

Originally posted by DaveBaumann
I think the real issue here is not the 3DMark performance, but whether the optimisation guidelines they have reiterated to the press on two different occasions now are actually real or not. Upon first looks it would appear that the 52.16 violates all three of their guidelines in 3DMark. Are they serious about these guidelines or are they just paying lip service to appease the press?
Ooooh, OOOOH!!! I know this one, it's just paying lip service to appease the press...isn't it?

Although I think you'd have to make that "the gullible press" too... ;)

[Sickness]

Benchmark specific optimizations are wrong where 3DMark is concerned. I agree with Dave on this. (Err, I think.) If FutureMark was not able to stifle the cheating they would not have the semblance of being objective in their performance comparisons. It is pretty clear that if you are not applying optimizations that do not work across the board for all DX9 applications that you are making performance improvements for a specific app. In this case the app. is meant to help end users determine general performance on a variety of software that uses DX9 API. Unless nvidia is creating these "optimizations" for all DX9 apps it isn't likely that you are going to get consistent results. In short the "optimizations" that nvidia made specifically for 3Dmark 2003 are totally unethical. But what is worse is that these optimizations change the output so that not only are the performance numbers unqualified but also the developers desired product is distorted, which is another no-no. To answer the question that Dave presented with regards to nvidia playing lip service the answer is most definitely ....YES, in my opinion of course.

[lukar]

3DMark03 uses DX 9.0API, and that's why r3xx doesn't suffer. But, NV3x is a different story, and in order to perform DX 9.0 applications as well as ATI card does, they have to hack and 'optimize' everything. People do not want to admit a one thing, NV3x chip is a garbage. Still, they are buying it!

Futuremark and Nvidia will play the game called 'the cat is chasing the mouse' till Nvidia release NV40. After that, NV3x won't have any support in the sense of 'cheating', and a dozens of users can suck.

The real punch in the face would be if ATI release the card which is going to kick NV40 as r3xx kicks NV30. :angel:

[Sickness]

Originally posted by lukar
3DMark03 uses DX 9.0API, and that's why r3xx doesn't suffer. But, NV3x is a different story, and in order to perform DX 9.0 applications as well as ATI card does, they have to hack and 'optimize' everything. People do not want to admit a one thing, NV3x chip is a garbage. Still, they are buying it!

Futuremark and Nvidia will play the game called 'the cat is chasing the mouse' till Nvidia release NV40. After that, NV3x won't have any support in the sense of 'cheating', and a dozens of users can suck.

The real punch in the face would be if ATI release the card which is going to kick NV40 as r3xx kicks NV30. :angel:

What nvidia is selling tons of is the 5200, which is a piece of "garbage" no question. The R300 was a "punch in the face", if ATi can top the NV40 it would be more a kin to "kicking them when their down." :cool:

[sebazve]

Originally posted by Sazar
has FM come up with any formal agreement as to what will happen if there are continued violations or perceived violations Dave?

afaik there is no stated actions to be taken in such a case but i could be mistaken...

they should ban the ****ing drivers end of story what happens when you cheat on a online game you get ****ing ban
:nutkick:

but problably half nvidia owners would be pissed off cause they wont be able to post their results, so FM would be in deep **** cause many peope is gonna stop using 3dmark so its bad bussiness.

i guess they should stop bitching or do something or we are gonna have 1000 patches trying to stop nvidia from cheating :rolleyes:

[vandersl]

The more I think about it, the more I think the problem FutureMark has is that people can see their benchmark.

Their business model kind of depends on having a benchmark with eye candy - they want end-users to download it, use it, and care about their scores. That is what makes the benchmark a powerful tool in selling graphics chips (to both end users and OEMs).

However, it seems that since the output is 'visual', people insist that 'close enough' is acceptable. If they can't see the difference in the output, then they don't care. This attitude allows IHVs to cheat on the benchmark, making it invalid for the very purpose it was intended.

Other synthetics don't seem to be treated the same way. I've never seen anyone make this type of argument about any other benchmark with a quantitative output.

If 3DMark was just a benchmark that computed numbers as an output, instead of colors, would anyone really argue that generating incorrect but 'close' numbers is OK? Really, I'd like to know.

[lukar]

ChrisW said here that the score for NV3x now drops in the 4000+ range, not in 3xxx range like before. So, I guess that Futuremark approved some of the 'optimizations' by Nvidia. I assume that Nvidia goes even further, something what Gabe mentioned for HL2 scenario.

The score that NV38 produces using the latest drivers in patched 3DMark03 which is in 4xxx range, was done by using FP16. I think Futuremak allowed it, and that's why we se 4xxx range score, but not 3xxx.

But the truth is they are using FX12, and that's unacceptable for Futuremark.

I don't blame Nvidia. They have to do what they have to do, to make $500
card to perform like supposed to, otherwise they are screwed. It's all about business and money, but not about consumers, and all we know that NV3x represents TITANIC...