Futuremark releases 3DMark03 patch - alleges NVIDIA cheats

[MrWho]

It's almost that time that comes every year when I must start thinking on upgrading my graphics card. I've been an nVidia customer for some time (TNT2, Geforce, and now my GF3 Ti500), but now the choice's much harder.

For some time now (and not because of this, though it helped) I've been considering changing over to ATI. I'm still afraid of their drivers (Although I've read they got much better than what they used to be), but the FX5800 doesn't inspire much confidence to me either (the noise, the screensaver bug, the benchmarks when compared to ATI's cards, and now this).

okay, you guys tell me ATI cheated too. What's that supposed to mean? Should I think "well, if ATI cheated too, then it's okay!"? The fact that ATI "only" cheated for little more than 1% makes me think if it wasn't just a fluke, a piece of bad programming in their drivers. Being true, that's bad because it shows that they're not capable yet of doing top-notch drivers, but it's good because it may prove that they didn't intend to do it. Now I know that nVidia has some good programmers writting their drivers (nVidia is to me the best driver supplier in the whole hardware department, not just gfx cards), which implies that whatever "cheats" are found inside their drivers (and, even worse, 29% cheats) can't be there by accident.

I change my mind easily when I feel I'm being played for a sucker. I was a 3Dfx fan until the advent of the Voodoo3 series - at that time I changed over to the TnT2. Now maybe that time is comming again, I'm still not sure. But one thing I'm sure: I'm an ex-nVidia fannatic - now I just plain like them, but I won't jump in their defence so easily.

[Behemoth]

Originally posted by Onde Pik
They changed the excution order of instructions, as I pointed out with an example this can be done on ANY architechture to improve performance without changing the end result 1 iota. And what are you talking about with the next driver??? How do you know they wont? Got a link to verify this?

If you dont grasp what is going on, then plz stop replying.
instead of letting the PS2 shader itself doing the flow of control, ati manually tells GT4 sky and water shaders how to shuffle instructions, hows that different to inserting clip plane to tell the video card ignore some of the workload, in this case ati is telling the video card to ignore some of the decision makings to get higher score, because the shaders were not doing full PS2 work.
the difference is nvidia fails to maintain visual output while ati succeeds to maintain it. but futuremark was not trying to measure how good PS2 could be in that particular case, the siturational modification was not what futuremark was trying to measure. hence ati is gonna to remove it in next driver, ati *helps* the score by hinting the shaders how to shuffle instructions.


"The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST."

[Hanners]

Originally posted by Behemoth
instead of letting the PS2 shader itself doing the flow of control, ati manually tells GT4 sky and water shaders how to shuffle instructions, hows that different to inserting clip plane to tell the video card ignore some of the workload, in this case ati is telling the video card to ignore some of the decision makings to get higher score, because the shaders were not doing full PS2 work.
the difference is nvidia fails to maintain visual output while ati succeeds to maintain it. but futuremark was not trying to measure how good PS2 could be in that particular case, the siturational modification was not what futuremark was trying to measure. hence ati is gonna to remove it in next driver, ati *helps* the score by hinting the shaders how to shuffle instructions.

I would say Pixel Shader 2.0 performance makes up a large proportion of how GT 4 performs.

Going by what Tim Sweeney of Epic has to say about things, what ATi did was a valid optimisation, but nontheless seeing as it is only implemented in the specific case of 3DMark 2003 GT4 it should be removed.

The whole point of a benchmark is that it runs 100% how the developer intends it, so even shuffling instructions is a no-go in my book.

[Behemoth]

Originally posted by Hanners
I would say Pixel Shader 2.0 performance makes up a large proportion of how GT 4 performs.

Going by what Tim Sweeney of Epic has to say about things, what ATi did was a valid optimisation, but nontheless seeing as it is only implemented in the specific case of 3DMark 2003 GT4 it should be removed.

The whole point of a benchmark is that it runs 100% how the developer intends it, so even shuffling instructions is a no-go in my book.
wow cant believe you agree with me on this one :D

[Morrow]

I'm wondering if FutureMark would not have written something about the 8% drop in performance in GT4 using ATI cards, had ATI ever officially admitted that they also replaced shaders programs in 3dmark03? Probably not.

Shows how trustworthy any company nowadays is. Why did ATI give their official statement on 3dmark03 just one day after nvidia has been proved cheating, why not weeks before?

Why are nvidia's shader optimizations called cheats and ATI shader optimzations legitimate, when FutureMark wants to test standard performance with standard DX calls and not specifically optimized routines for any hardware vendor?

Why is ATI removing those optimizations in the next Catalyst release even if they think that those optimizations are legitimate and do not question the fairness of this "synthetic benchmarking"?

This all shows again how obsolete benchmarks à la 3dmark03 have become even if FutureMark is investigating strange driver behaviours and making changes to the 3dmark03 code to prevent cheating. Doesn't this show how easily any company can optimize their drivers to fake their results in any way they want.

Going back to use my videocard for what I have bought it in the first place.... playing games

[Hanners]

Originally posted by Morrow
I'm wondering if FutureMark would not have written something about the 8% drop in performance in GT4 using ATI cards, had ATI ever officially admitted that they also replaced shaders programs in 3dmark03? Probably not.

I believe they would, yes.

Originally posted by Morrow
Shows how trustworthy any company nowadays is. Why did ATI give their official statement on 3dmark03 just one day after nvidia has been proved cheating, why not weeks before?

I get the impression ATi hadn't even considered switching the order of a few shader instructrions around as a cheat until this whole situation came to light.

Originally posted by Morrow
Why are nvidia's shader optimizations called cheats and ATI shader optimzations legitimate, when FutureMark wants to test standard performance with standard DX calls and not specifically optimized routines for any hardware vendor?

Simple - nVidia are replacing FutureMark's shaders wholesale with their own, and suffering reduced image quality as a result. ATi are using FutureMark's shaders, but switching the order of some of the instructions so it runs faster on their architecture.

Originally posted by Morrow
Why is ATI removing those optimizations in the next Catalyst release even if they think that those optimizations are legitimate and do not question the fairness of this "synthetic benchmarking"?

Because at the end of the day it isn't about what ATi think, it's about what FutureMark and the consumers think. In general people feel that it is a cheat, therefore ATi will remove them from their drivers.

[Madstrom]

What I find disturbing is the difference in responses from both companies. ATI answers about their "optimizations" and plans to remove them, nVidia responds "WAAAAAAAAA!WAAAAAAAAA! They want to make us look bad.WAAAAAAAAA!":mad:

This makes me wonder about the other benchmarks. We know that they can be "optimized." Just like the ATI Quake III cheat!

That cheat kept me away from ATI for a long while. Heck I didn't even try ATI until I found an 8500 for $106.00.

Now I feel the same about nVidia:(

Luckily all my previous nVidia cards were the good, solid ones (TNT, TNT2, GeForce, GeForce 2 GTS, GeForce 3 Ti 200, GeForce 4 Ti 4400) that kicked ass without cheats.

[chrismm28]

Originally posted by Behemoth
instead of letting the PS2 shader itself doing the flow of control, ati manually tells GT4 sky and water shaders how to shuffle instructions, hows that different to inserting clip plane to tell the video card ignore some of the workload, in this case ati is telling the video card to ignore some of the decision makings to get higher score, because the shaders were not doing full PS2 work.
the difference is nvidia fails to maintain visual output while ati succeeds to maintain it. but futuremark was not trying to measure how good PS2 could be in that particular case, the siturational modification was not what futuremark was trying to measure. hence ati is gonna to remove it in next driver, ati *helps* the score by hinting the shaders how to shuffle instructions.
No the ATI drivers were rendering the scene exactly as the 3dmark code wanted, just in different order. This can be done in any situation, including real games. Its what optimisations are, from any company. Nvidias drivers actually replaced the 3dmark code completely to create a similar looking scene that was easier for the nv3X to render, but it was not identical.
If you had an exam to do you might not do all the questions in order, but come back to some later.. this is ATI.
If you had an exam but found the questions to hard, you would change them so that the answers were similar, but the question itself was easier... this is nvidia.

[Madstrom]

Sorry if this is a repost, but here's FutureMark's take on the nVidia/ATI difference. This came with my email and had the Q&A at the bottom.



Q: Do scores change with the patch?
A: Yes. Although the actual workload in the benchmark is not changed at all,
we have noticed substantial performance score changes in NVIDIA hardware and insignificant performance change e.g. with ATI hardware. Thus, only build 330 scores should be used from now onwards.

[dante_uk]

Really, what's the point?

You look at the benchmarks for the following:
Q3 Engine ( at least a dozen licenced games and more coming )
Unreal Engine ( Lots of licenced games and more coming )
DooM 3 ( Doom3 , Quake4, unknown title by HumanHead Studies)
3dMark2003 ( none, nothing, petty tech demo )

Compare top Nvidia card with top ATI card.
Currently:
Nvidia wins in Quake3, Unreal, Doom3
ATI wins in 3dMark2003

Okay now which card do I buy ?

How can those results be fair and true?
None of those game engines are DirectX9, but then only one game test in 3dMArk2003 is actually using any DirectX9.

If the top ATI is better than the top Nvidia card why don't ANY other benchmarks show that?

What's Cheating ?
If there's not code to make these game engines run faster in the drivers from both companies then why the hell not - We all play games using these engines!!
I for one want the best performance and best images I can get.

3dMark2003 claims to be about apples-to-apples comparisons. What's the point?
I Write a DirectX function to spin a cube. It works, it shows me a FPS score. One card runs it faster than another.
Is the faster card going to play quake 3 better and faster then the other card? Would anyone be willing to bet money on which card plays unreal 2 better and faster based on how my spinning cube runs?

Then you have the fact that 3dMark is only DirectX.
Most of the games I play use the quake 3 engine ( opengl ) plus I play IL2 using the OpenGL renderer because it's faster, I play UT2003 using the OpenGL renderer because it's faster.
What should I use to see who has the best OpenGL support on their cards?
Answer: easy, I run Q3 and UT2003 !!

Nvidia is only guilty of wasting time bothering to get a better score with 3dMark2003.
Anyone stupid enough to put their faith in the performance of this program deserves what they get.

[digitalwanderer]

Originally posted by dante_uk
Compare top Nvidia card with top ATI card.
Currently:
Nvidia wins in Quake3, Unreal, Doom3
ATI wins in 3dMark2003

Yeah, nVidia wins if you leave AA & AF off...if you turn any of those on the ATi card toasts 'em. I don't know about you, but if I'm paying HALF-A-GRAND for a a video card I'm kind of expecting to use AA & AF.


If the top ATI is better than the top Nvidia card why don't ANY other benchmarks show that?


Again, turn on AA & AF and see the difference both in image quality & speed...nVidia just can't compare right now. :)



What's Cheating ?

nVidia, I thought we had covered that one already. :rolleyes:


3dMark2003 claims to be about apples-to-apples comparisons. What's the point?

A fair and honest head-to-head comparison of a cards features and performance, some of us like that. ;)

[legion88]

Originally posted by dante_uk
Really, what's the point?

You look at the benchmarks for the following:
Q3 Engine ( at least a dozen licenced games and more coming )
Unreal Engine ( Lots of licenced games and more coming )
DooM 3 ( Doom3 , Quake4, unknown title by HumanHead Studies)
3dMark2003 ( none, nothing, petty tech demo )

Compare top Nvidia card with top ATI card.
Currently:
Nvidia wins in Quake3, Unreal, Doom3
ATI wins in 3dMark2003

Okay now which card do I buy ?

How can those results be fair and true?
None of those game engines are DirectX9, but then only one game test in 3dMArk2003 is actually using any DirectX9.

If the top ATI is better than the top Nvidia card why don't ANY other benchmarks show that?

What's Cheating ?
If there's not code to make these game engines run faster in the drivers from both companies then why the hell not - We all play games using these engines!!
I for one want the best performance and best images I can get.

3dMark2003 claims to be about apples-to-apples comparisons. What's the point?
I Write a DirectX function to spin a cube. It works, it shows me a FPS score. One card runs it faster than another.
Is the faster card going to play quake 3 better and faster then the other card? Would anyone be willing to bet money on which card plays unreal 2 better and faster based on how my spinning cube runs?

Then you have the fact that 3dMark is only DirectX.
Most of the games I play use the quake 3 engine ( opengl ) plus I play IL2 using the OpenGL renderer because it's faster, I play UT2003 using the OpenGL renderer because it's faster.
What should I use to see who has the best OpenGL support on their cards?
Answer: easy, I run Q3 and UT2003 !!

Nvidia is only guilty of wasting time bothering to get a better score with 3dMark2003.
Anyone stupid enough to put their faith in the performance of this program deserves what they get.

What a weak commentary.

I'll keep it simple so try to keep up.

Performing well in Quake 3 benchmarks does not translate to performing well in DirectX games. Duh! So why does it matter that 3DMark is "only DirectX"? Duh! Quake 3 is only OpenGL. Duh, again. Using logic goes a long way. Try it.


As I pointed out already, "opti-cheating" to improve static benchmarks (like timedemos in Quake 3 or "synthetic" benchmarks like 3DMark2003) does not translate to improved performance in real-time usage. Duh! So all "opti-cheating" does is mislead the public into believing that their real-time frame rates would go up alot when it, in reality, it won't.