Futuremark releases 3DMark03 patch - alleges NVIDIA cheats

[Clockwork]

3DMark03 Patch 330 Released
Fri 2003-05-23

Saratoga, California USA – May 23rd, 2003

3DMark®03 incorporates many ground breaking new features and technologies and sets the standard for easily and objectively measuring and comparing modern PCs’ 3D graphics performance. Recently, there have been questions and some confusion regarding 3DMark®03 results obtained with certain NVIDIA drivers.

We have now established that NVIDIA’s Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark®03. We have just published a patch 330 for 3DMark03 that defeats the detection mechanisms in the drivers and provides correct results.

Download the 3DMark03 Patch 330 here:

http://www.futuremark.com/download/?3dmark03patch330.shtml

A detailed audit report of the drivers and 3DMark03 is available at:

http://www.futuremark.com/companyinfo/3dmark03_audit_report.pdf

As with all Futuremark’s benchmarks, 3DMark®03 has been created in cooperation with the major PC industry players to provide users with the best possible and most reliable set of diagnostic tools.

Futuremark is regarded as the driving force that pushes the industry to raise the bar with respect to graphics performance and features. We are committed to deliver performance analysis in an accurate, fair and consistent manner. 3DMark®03 continues in that tradition.

Additional Information Please visit http://www.futuremark.com/products/3dmark03 for additional 3DMark®03 related information. 3DMark®03 is available for free public download at http://www.futuremark.com/download/?3dmark03.shtml

About Futuremark® Corporation
Futuremark® Corporation, formerly known as MadOnion.com™, is the leading provider of computer performance analysis software and services. Futuremark® is known around the world for its benchmark products, including the 3DMark® Series and PCMark™2002 (with more than 30 million copies distributed worldwide) and value-added services powered by a database of over 5 million real life benchmarking results. Futuremark® has offices in Saratoga, California and Helsinki, Finland. For more information, please visit http://www.futuremark.com.

© 2003 Futuremark® Corporation. 3DMark® and PCMark™ trademarks and logos, Futuremark® character names and distinctive likenesses, are the exclusive property of Futuremark Corporation. DirectX® is a registered trademark of Microsoft Corporation. All other trademarks are property of their respective companies.

Company Contact:

Futuremark Corporation
Phone: +1-408-517-0131
Mr. Tero Sarkkinen
Executive Vice President of Sales and Marketing
Email: tero@futuremark.com



http://www.futuremark.com/news/?newsarticle=200305/2003052308#200305/2003052308

Special thanks goes out to Nebuchadnezzar! :D ;)

[Nebuchadnezzar]

I didn't think you would take it seriously :eek2:

[DSC]

From the audit PDF:

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on
this new build of 3DMark03. The drop in performance on the same test system with a Radeon
9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to
8.2% difference in the game test 4 result, which means that the test was also detected and
somehow altered by the ATI drivers. We are currently investigating this further.

HAHAHAHAHA, so ATI CHEATS also. Good work. :p ;) :D

[deejaya]

This is interesting. If you read the PDF, you find that although not as great a difference, the ATI drivers were found to be "detecting" 3dMark too.

Makes you feel all warm inside when you know the only alternative (to all those people banging on about trust) is Matrox. :angel:

[Mariner]

Yup - ATI cheated by 1.9%
NVidia cheated by 24%

I suppose NVidia's driver team has always had the reputation of being better than that of ATI! :D

[PreservedSwine]

Who's going to download the patch run the tests? I'd like to see the new results for both cards.

I never, ever thought I'd ever say this, but Great job futuremark! :D

[tertsi]

Copy from Futuremark's report:

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Busted!

[DSC]

At least we can now happily be in the one big family of cheaters, whether you're an Nvidiot or fanATIc..... :D

GROUP HUG GROUP HUG!!!!! :angel:

[PreservedSwine]

Originally posted by DSC
At least we can now happily be in the one big family of cheaters, whether you're an Nvidiot or fanATIc..... :D

GROUP HUG GROUP HUG!!!!! :angel:

I think you've missed the point. Cheating is bad, & that means even if everyone is doing it. ATI owners ahould feel cheated as well.

This doesn't make it OK, and way to go Futuremark for exposing any cheats!

[digitalwanderer]

Originally posted by tertsi
Copy from Futuremark's report:

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Busted!

No, no...that one is a legitimate bug. ;) :p

Gotta say it, nice job to futuremark for catching both of 'em (if ATi's is an actual cheat, I was joking 'bout with the legitimate bug bit. ;) ) and I hope they keep it up. :)

[c4c]

Originally posted by deejaya
This is interesting. If you read the PDF, you find that although not as great a difference, the ATI drivers were found to be "detecting" 3dMark too.

Makes you feel all warm inside when you know the only alternative (to all those people banging on about trust) is Matrox. :angel:

Matrox probably just hasn't been important enough to scrutinize yet. Besides, they have enough problems as it is :)


To the topic at hand:

I have to say I am VERY impressed with this statement from Futuremark. I hope they continue to release reports like this about cheats/unexplainable performance. This will do alot towards making 3DMark a worthy benchmark, instead of a nice demo that shows some numbers at the end. (which is what all the previous versions have been to me)

Please please please keep watching out for stuff like this guys. It is interesting to see that Ati might also have their hand in the cookie jar.

[Nebuchadnezzar]

Rofl you people trying to tell that that 1.9% difference on the ATI cards justifies the 24% of the Nvidia's? :rolleyes:

[DSC]

Yes I understand, that was meant as sarcasm.

I hope this will prove once and for all to ATI defenders that yes, your company CHEATS also. :rolleyes:

[Darren555]

Yeah bad week for nvidia.........first news that Dawn prefers ATI & now this!! ;) :p

[Nebuchadnezzar]

First of all let's wait for Futuremark to insestigate on the ATI drivers before saying anything about cheating. :rolleyes:

[Kombatant]

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Well, if you look at it from a different perspective, it could mean that the ATi driver guys are so incompetent that their cheats don't even produce good results LOL :cool: :D

[Toastywheatman]

The performance drop concerning ATI is not yet confirmed to be cheating until it can be shown that it in some way reduces image quality or does not fulfill the precise instructions sent to the driver by 3DMark2003, as was shown with the Nvidia drivers.

It may be that ATI has a 'game-specific' optimization in the mother nature test that does not take any artificial short cuts which reduce work load. THIS IS VERY IMPORTANT. If the ATI drivers are still doing EVERYTHING that the 3DMark2003 application asks, then it is not cheating.

If this is the case for ATI, I hope that they come out and say it so that they can dispel any doubts.

-Toasty

[Skinner]

Originally posted by PreservedSwine
Who's going to download the patch run the tests? I'd like to see the new results for both cards.

I never, ever thought I'd ever say this, but Great job futuremark! :D

I'm downing 3dmark03 now (yes, never run it, ok one time, but all I care are games ;) )

[rokzy]

Originally posted by Nebuchadnezzar
Rofl you people trying to tell that that 1.9% difference on the ATI cards justifies the 24% of the Nvidia's? :rolleyes:

yep 1.9% could easily be a bug or an error. either way I'm glad this sort of thing is being discovered because at the end of the day I want to see accurate results.

nvidia's 24% is just ridiculous. it reminds of when the U.S. was helping rig elections in Vietnam before the war. the U.S. suggested faking a win of about 98%, but they went for a 100% victory, which was obviously just total BS.

[digitalwanderer]

If ATi's 1.9% is actually a driver detection routine than I intend to have some words with some ATi folks...albeit a lot more pleasant ones than with nVidia. (Hey, I know ATi will listen to me without shouting! ;) )

[deejaya]

Originally posted by Nebuchadnezzar
Rofl you people trying to tell that that 1.9% difference on the ATI cards justifies the 24% of the Nvidia's? :rolleyes:

Are you trying to justify the 1.9% difference as being acceptable in light of nVidia being caught? The report clearly states "This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers."

Cheating is cheating is cheating, the same applies for any company. It should not be done, no matter how large or small the gains, a cheat is a cheat. It's like saying "Ah, I'm only using a wallhack though, at least I'm not using an aimbot"

You are right to say to wait, but take a little look around this forum (and others) to see how long people waited before calling nVidia cheaters.

[c4c]

Originally posted by Nebuchadnezzar
Rofl you people trying to tell that that 1.9% difference on the ATI cards justifies the 24% of the Nvidia's? :rolleyes:

If anything it highlights just how bad Nvidia is cheating.

They mention that the 1.9% score difference is almost entirely on a 8.2% score difference in game test 4. An 8.2% difference is nothing to sneeze at, but now I think about the 24% difference in Nvidia scores :eek:

Of course it remains to be seen if Ati is cheating or not. But it is pretty clear Ati is doing SOMETHING to detect 3DMark and make it faster..At least for the one test.

IMO this should come as a surprise to nobody. This is the first REAL scrutiny I have seen from 3DMark, so it makes me wonder how bad the cheating is on previous versions (from both companies). After all, if they can get away with it, they will do it.

[Sazar]

fyi... from the driver reports from 3dmark03...

In our testing, all identified detection mechanisms stopped working when we altered the
benchmark code just trivially and without changing any of the actual benchmark workload. With
this altered benchmark, NVIDIA’s certain products had a performance drop of as much as
24.1% while competition’s products performance drop stayed within the margin of error of 3%. To
our knowledge, all drivers with these detection mechanisms were published only after the launch
of 3DMark03. According to industry’s terminology, this type of driver design is defined as ‘driver
cheats’.

note the +/- 3% variance that they have listed... v/s the 1.9% variance shown by the ati driver set overall..

however with the 8 odd percent variance in gt4... there may well be some optimization going on... thus far no one has reported on it..

will be interesting to read the reports on the cat 3.4 series to see just what the 'optimizations' were..

but those same people who were deflecting criticism of nvidia's cheats are now taking it upon themselves to bash ati for a variance of 1.9% v/s well over 20% for nvidia's driver hacks... it is a little hard to understand why this is so...

[schuey74]

Give a lot of credit to Futuremark for not using this as an all out attack on Nvidia. Given Nvidia's very outspoken discrediting of 3DMark03, it would have been very easy to get drawn into that game. Instead they chose to learn from Nvidia's mistake and not try to hide something that may come up later. By releasing the patch and making public the fact that both Nvidia & ATI had cut some corners in their tests they are at least making a decent effort into making 3DMark03 a more accurate benchmark.

I'm not saying that 3Dmark is now the end all, definitive benchmark, but I like the way Futuremark handled the issue.......discreet & fair.

[Moose]

I think this is a GREAT move by Futuremark.

They FOR SURE have caught Nvidia cheating and they may have caught ATI as well and will be investigating further.

This is good for the entire computer industry as well as all computer enthusiasts.

We need to have valid benchmarks so we know how good the products we are going to buy, REALLY are.

I think Futuremark has done a good service to help legitimize the benchmarking process.

Kudos.

Although I think their work in this area will need to be ongoing to prevent future cheating.