NVIDIA's offical response to Futuremark

[gordon151]

Originally posted by Morrow
nope, I didn't miss both the extremetech and the FM audit report but unfortunately most people here seem to miss the big picture about this joint statement and what it means for the future:

When 3dmark03 was released, nvidia had quit the beta member team because they did not agree with FM's philosophy about benchmarking "standard performance". Nvidia knows that (IQ loss-less) optimizations are an important factor for demonstrating the full power of GeForce cards. All important games (especially ID games) make full use out of those optimizations but now comes FM and released a "gaming benchmark" as they call it but only relies on standard DX code which is never used that way in games... of course nvidia quit

Then came FM's statement a few days after the release of 3dmark03 that ANY change in the driver code to increase performance in 3dmark03 is not legitimate. Nvidia again couldn't understand how FM wants 3dmark03 to be the benchmark for future game performance when they code their engine exactly the way future real game engines will never be coded. So nvidia decided to boycott FM by saying that 3dmark03 is not reliable because of easy to make driver optimizations to synthetically increase the 3dmark03 score and they even went as far as demonstrating it in their 44.03 drivers.

But this whole optimization/cheat disbute now got finally a postive turn in a way that allows nvidia and all other graphiccard manufactures to include driver optimization (like done in real games) in future versions of the popular 3dmark benchmark (not yet including 3dmark03) making this bench or more real-life indicator that it currently is.

This means that nvidia doesn't need to hide anymore optimizations in their drivers, no longer needs to replace shaders effecting IQ but can go straight to FM and help them on how to accomplish much faster code with the same IQ. It will be all more easy and transparent for everyone. It's a clearly win/win situation.

Why do you keep comparing 3DMark03, which is based on a standardized format with no vendor specific extensions to an OpenGL format which uses vendor specific extensions? Why do you expect 3DMark03 to be a vision of Doom III performance which by Carmacks own accord, doesn't make full use of all the new features in the current DX9 hardware. It's just a case that Doom III requires a DX9 card purely on the merits that it just needs the raw processing power to calculate all the shadowing effects and complex textures.

Furthurmore ATI was the only company of the two that used an optimization in 3DMark03 that had no effect on IQ at all since it was just an optimization of an existing routine. Two highly influential developers have gone on the record as saying that such a thing is an extremely valid optimization. What they didn't say was a valid optimization was anything that implemented a loss of visual quality from the intended output and in the case of synthetic benchmarks anything that reduces the workload of one specific card.

It would also be nice to point out that benchmarks are and should be solely a tool for users. How can one accurately be sure of any benchmarking tool where the manufacturer of said product being benchmarked has an active hand in exactly what is being done in the program to gauge his card? Before this high level members had an influence of "suggesting" what was going into the benchmark, but now it looks like they (or more specifically Nvidia and not the users) have a more direct hand in determining what is fair or not towards their product.

I for one can't trust any timedemo from now on from both ATI and Nvidia (more so for Nvidia absolutely) and subsequently will be more inclined to make buying decisions from actual benchmarks of playing performance like that which were done in nV News. Nvidias actions have so plainly caused such a scenario to be the case for quite a lot of people.

[digitalwanderer]

Originally posted by Solomon
Um... The clip plane is not an optimization! As soon as you can see that the sooner you will realize that nVidia cheated and is buying it's way out of the paper bag.
I wouldn't even waste your time arguing with him. Either he's a totally brainwashed fool who actually believes nVidia's version of reality or he's just looking to provoke a reaction. Whichever it is, I'm a little tired of wasting me time on the clueless and the attention seekers.

This whole thing just really sucks and makes me feel all dirty about something I love, very unhip. :(

[dsessom]

With all the allegations flying, it's hard to tell just exactly what is truth and what is not, but here are a few thoughts...

nVidia has never backed 3DMark'03 since it was released, although they were involved with development - almost right up to the finish. But since it's public release, nVidia has not supported the benchmark for whatever reason, probably because their FX5800 was getting lower scores than the 9700 Pro, but I digress...

I think that Futuremark is trying their best to have a reliable, respectable product but they have gotten muddled in "application optimization" accusations from both ATI and nVidia because both companies want the fastest product and they are pretty neck and neck right now.

So, here is the real question:
Are nVidia and ATI's "Optimizations" so dramatic that you consider it cheating? OR - are they darned ingenious pieces of code that have potential to be used in the real world? ie:
if nVidia's drivers can "optimize" for 3DMark03.exe, then surely they could do the same for a number of other specific games that would benefit.

I don't really think Futuremark did anything to deserve all the slamming they have gotten. From my point of view, ATI and nVidia are the ones to blame if anyone at all - but that just goes back to the paragraph above.

So, am I just way out in left field? Or perhaps just too optimistic??

[Dazz]

This got me.......
Translation VIA HardOCP: FutureMark reneges on previous statements and confirms NVIDIA was not cheating on their benchmark and NVIDIA will not take a legal action against FutureMark that would bankrupt them. All about the $, but that is just our opinion. Still this does not change our thoughts on the FutureMark and their benchmark. Editorial coming...
Looks like an Microshaft company, Win at any cost :rolleyes:

[Solomon]

Originally posted by digitalwanderer
I wouldn't even waste your time arguing with him. Either he's a totally brainwashed fool who actually believes nVidia's version of reality or he's just looking to provoke a reaction. Whichever it is, I'm a little tired of wasting me time on the clueless and the attention seekers.

This whole thing just really sucks and makes me feel all dirty about something I love, very unhip. :(

Yeah, I came to that conclusion too. Maybe he's Derek Perez in disguise? LOL

Regards,
D. Solomon Jr.
*********.com

[Solomon]

Originally posted by dsessom
So, here is the real question:
Are nVidia and ATI's "Optimizations" so dramatic that you consider it cheating? OR - are they darned ingenious pieces of code that have potential to be used in the real world? ie:
if nVidia's drivers can "optimize" for 3DMark03.exe, then surely they could do the same for a number of other specific games that would benefit.

I would consider it cheating if they don't follow the guide lines of optimizing their code. The clip plane was the nail in the coffin. Like I said awhile ago, if they want to optimize thier drivers that's fine, but when it comes to missing graphics and all this mess up. Then it's cheating. The "Cheating Drivers Hidden Dragon" picture sums up everything.

Regards,
D. Solomon Jr.
*********.com

[Hellbinder]

Well then ATi did not cheat either. Further it is now open Season on doing whatever you want to 3dmark03.

Question..

Didn't B3D prove that they used LOWER than Dx9 standard Rendering to achieve some of their speed boost??? How can Futuremark say that using FX12 etc in a Dx9 benchmark is legit???

I just dont understand this.

[MrNasty]

Futuremark are sellouts. Nuff said.

[Morrow]

Originally posted by Solomon
So from this you are saying you rather have an nVidia optimized benchmark then a standard benchmark? There is a reason why it's a standard benchmark. nVidia can optimize for the benchmark. Hell no one is saying they can't. They can't cheat is what it's all about. The "Cheating Drivers Hidden Dragon" picture says it all. The clip plane says it all.

From what I'm getting from you is that it's o.k. to have all that not show and still be legit? You got to be kidding me.


no, you still don't get me. Let me put it this way, I don't like any benchmark from which company whatsoever. My opinion on benchmarks is that is only for the bragging rights which seems quite popular these days.

Considering decreased IQ in 3dmark03. You didn't understand my previous post. I said that nvidia probably did this cheating/optimization deliberately to show the weakness of benches (clip plane or not clearing back-buffer) for example. Those "optimizations" are hopefully not the kind of optimizations nvidia will now work on future games or the next 3dmark. The clip-plane was just one thing nvidia to demonstrate how easy it is to manipulate benchmark results.

The optimizations done in games however are a different thing and since the first Q2 nv optimizations we all know that you get a speed increase from them WITHOUT a IQ decrease and exactly those optimizations is what separates 3dmark from "real-world" performance. Did I finally made myself clear?

I do not endorse cheating in games in regards of IQ nor am I blindfully believing nvidia whatsoever but I believe that nvidia succeeded in making it pretty clear that optimizations matter these days, that the graphiccard industry has changed quite a bit and if you want a "fair" benchmark, you need to put the tested hardware in the best light possible and not in just an odd position where it never was designed to be!

Again, I'm not talking about the "optimizations" nvidia did in 3dmark03 or ATI did in Quake3 about which people should get over now, but I'm talking about real optimizations like they happened in all importants games the last 10 years.

Let's burry Quack, 3dmark and benches in general and move on to the real thing which matters the most and hopefully the reason why people buy their shiny new graphiccards: Games and applications like 3ds max.

[DivotMaker]

I guess I have a problem with FM's position on this in that they will now allow IHV-specific optimizations.

Last time I checked, the whole purpose for creating a benchmark is to compare products on a level playing field. The problem is establishing a "level playing field". This is likely never going to happen because the IHV's insist on differentiating their products. Therefore, any time the benchmark does not favor any of the "specific features" that differentiates each product, then each company is going to be screaming about it. The point I am making is in reference to nVidia's decision to use FP 32 instead of FP 24 as is the DX9 standard in the high end shader.

If I understand correctly, the FP 24 standard for DX9 was estabilshed before or during design work for GF FX 5800. If so, I am not certain why nVidia would not change this for the 5900 unless it is essentially similar silicon and changing at the stage of production (Feb 03 til now) would have meant a re-taping of a new chip. I think it is likely that nVidia underestimated the horsepower needed to compete with FP32.

I guess what I am trying to say is that unless companies are REQUIRED to adhere to SPECIFIC requirements of identically classed cards through Direct X and or Open GL, then I am afraid you will see more incidents like this one which are not helping a "less-than-healthy" PC gaming industry. To my knowledge, ATI has complied to the DX9 standard. nVidia has not at least from the shader perspective.

I am very satisfied that ATI and nVidia are competing and trying to give us more features with every hardware release. However, unless each of them are forced to adhere to specific features with which to compare equally on a level playing field, then MEANINGFUL, LEGITIMATE, and ACCURATE benchmarking has a very long tough road ahead.

[Hellbinder]

ATI has complied to the DX9 standard. nVidia has not at least from the shader perspective.

No, Nvidia Has complied. Their Hardware is 100% Dx9 Complient in every way. The problem is they made some rather poor Design choices in the light of Raw Performance. They can Run Dx9 complient all they want. But, They did not include the hardware under the hood to compete at Full DX9 percision.

I want to point out again that FP32 is not any harder to do than FP24 *if* you have the hardware in place to handle the load.

Thus Nvidia Could run Full Dx9 all day long. But, if they do it all by the book with no *cheating* *hacking* *reducing Quality* *shortcuts* etc.. they will get spanked.

This is really a tempory issue for them though. As I am pretty confident that the Nv40 will have full power to the Phaser banks so to speak ;)

[gordon151]

Hellbinders comments making sense and offering unbiased useful information. Hrm, that's not good....

[Solomon]

Originally posted by BigBerthaEA
I guess I have a problem with FM's position on this in that they will now allow IHV-specific optimizations.

Last time I checked, the whole purpose for creating a benchmark is to compare products on a level playing field. The problem is establishing a "level playing field". This is likely never going to happen because the IHV's insist on differentiating their products. Therefore, any time the benchmark does not favor any of the "specific features" that differentiates each product, then each company is going to be screaming about it. The point I am making is in reference to nVidia's decision to use FP 32 instead of FP 24 as is the DX9 standard in the high end shader.

If I understand correctly, the FP 24 standard for DX9 was estabilshed before or during design work for GF FX 5800. If so, I am not certain why nVidia would not change this for the 5900 unless it is essentially similar silicon and changing at the stage of production (Feb 03 til now) would have meant a re-taping of a new chip. I think it is likely that nVidia underestimated the horsepower needed to compete with FP32.

I guess what I am trying to say is that unless companies are REQUIRED to adhere to SPECIFIC requirements of identically classed cards through Direct X and or Open GL, then I am afraid you will see more incidents like this one which are not helping a "less-than-healthy" PC gaming industry. To my knowledge, ATI has complied to the DX9 standard. nVidia has not at least from the shader perspective.

I am very satisfied that ATI and nVidia are competing and trying to give us more features with every hardware release. However, unless each of them are forced to adhere to specific features with which to compare equally on a level playing field, then MEANINGFUL, LEGITIMATE, and ACCURATE benchmarking has a very long tough road ahead.

Well said...

Regards,
D. Solomon Jr.
*********.com

[digitalwanderer]

Originally posted by gordon151
Hellbinders comments making sense and offering unbiased useful information.

I know!

"How can you tell when Hellbinder is REALLY pissed off?" ;)

[SmuvMoney]

Am I only one feeling this way after reading the FM/nV statement?

June 2, 2003 - The Day The Benchmark Died...

IMHO, nVidia's lawyers said to FM's lawyers, "Remember 3DFX?" and it just went downhill from there. Like 3DFX, they had (the potential for) a valid gripe/complaint/lawsuit. Unfortunately, FM didn't have enough cash to legally box with nV for years to come - I think they have a staff of 26 employees or something like that. I guess FM didn't want to go out like 3DFX, i.e., assimilation/destruction, so FM did what it could to keep itself alive. Unfortunately, FM's immediate/short term goal for survival may have opened/unleashed Pandora's Box when it comes to any and all benchmarks be they synthetic or ingame.

I mean how many game benchmarks are going to have dev tools like 3dmark03? I'd rather have the devs working on the GAME ITSELF to improve the actual game experience. That being said, what will stop IHVs from cheating in any game benchmark especially if the dev can't provide decent benchmark tools like FM did? Like FM or not, the benchmark attempt to create a level playing field to compare video cards even if the methods were not the absolute cleanest or best. With this statement, it looks like you can bend any benchmark to your will. Official/formal/review benchmarking as I see it now is completely boned...

[DivotMaker]

Originally posted by BigBerthaEA
To my knowledge, ATI has complied to the DX9 standard. nVidia has not at least from the shader perspective.


Originally posted by Hellbinder
I want to point out again that FP32 is not any harder to do than FP24 *if* you have the hardware in place to handle the load.


I see your point, incorrect wording on my part. What I meant to say is that for whatever reason, nVidia did not allow for FP24 in their hardware. I don't know the "why, when, where, or how" this occurred, but if nVidia did have FP24, then it is possible this entire issue could possibly have been avoided. I think FP32 looked glamorous at the time it was implimented, but I don't think enough was known at the time about:

a) how much raw power was going to be rquired to make FP32 a true advantage over FP24

b) how competitive R3xx would be at FP24

From what I am hearing, FP24 and FP32 are overkill for the games coming up this year and possibly even next year with respect to the amount of sheer horsepower to run those features as developers intend for them to be run. I guess this can be chalked up to the incessant competition and "one-up-manship" of competing in today's graphics market. At the end of the day, I feel this situation, as it is today, is far more confusing to the average consumer than it ever should be. Hell, if we don't know what to make of it, how is the average Joe Q. Public going to make and informed decision?

[DivotMaker]

Originally posted by gordon151
Hellbinders comments making sense and offering unbiased useful information. Hrm, that's not good....

Actually it is VERY good...

I have to say that I respect what he has said and how he has handled himself here recently. He and I had a PM chat and I think we understand each other much better.

[frenchy2k1]

Originally posted by SmuvMoney
Am I only one feeling this way after reading the FM/nV statement?

June 2, 2003 - The Day The Benchmark Died...


Actually, this is probably not such a bad thing. Now, instead of just spending our days running useless benchmarks to compete for bragging rights, we could return to playing games...

Seriously, as stated in the PR, all the other bench are using processor specific optimisations. The graphic chips are getting closer and closer to general purpose processors, with shaders used instead of fixed treatment. Intel wins in most benchmarks because they are using their extensions SSE2, do you see AMD complain? They won in FP before thanks to their architecture. Graphic is just becoming the same way.

And dont talk about 3DFX. They were the first to optimize this way: nobody remember the miniGL drivers? What do you think those were?

After all, if nvidia spends time to optimize their drivers on popular games, more power to them. Sure, they look better at UT2k3 Flyby, but the game benefits also. 3Dmark is only here for bragging rights! (and so is Quake3 now, with its 300+ fps...)

[Hellbinder]

From what I am hearing, FP24 and FP32 are overkill for the games coming up this year and possibly even next year with respect to the amount of sheer horsepower to run those features as developers intend for them to be run. I guess this can be chalked up to the incessant competition and "one-up-manship" of competing in today's graphics market. At the end of the day, I feel this situation, as it is today, is far more confusing to the average consumer than it ever should be. Hell, if we don't know what to make of it, how is the average Joe Q. Public going to make and informed decision?

This is the problem.

I hate to point this out but Percision has nothing to do with the horsepower required to run full DX9. I notice this very often, that people here generally have a slighty quirky View of what the real issues are. Based mostly imo on Nvidia PR statements.

ATi Can run every application now or in the Future at full 24 bit Float with no performance loss. Why?? becuase they have 8 Fully supported FP24 pilenines. That are built from the ground up to use *only* FP24. No more no less. Where slowdown comes in is in Shader instruction processing and all the possible processing that gets involved that may take multiple passes. FP24 and FP32 color has literally no overhead cost except for Tansistors. You put the Transistors on the chip to handle it and thats all that is required. This is not a Bandwidth issue, or a Fillrate issue.

Nvidia simply shot their foot off in the Number of 32bit registers they have to work with. In fact as I have been getting more informed lately on the Nvidia side of thigs what they are really doing is not even dropping to FP16 as they have similar Register support for FP16 and FP32. What they are really doing is Using Cg to recompile the shaders to use FX12 (Fixed Function 12 bit) in many cases.

[Zenikase]

The fact is that even if you have an unoptimized benchmark test, it will always favor certain products over others simply because of the way they are coded and compiled. Athlons and Pentium 4s have very different methods of fetching, translating, organizing, and executing data throughput. It's really almost impossible to make an apples-to-apples comparison of the two due to the way their architectures are designed to work. Athlons feature fully-pipelined execution units and the EV6 bus while the P4 is equipped with an extraordinarily long pipeline for scalability and the innovative trace cache, and all this goes without mention of the pipeline sequence and layout.

The same thing occurs in the video card industry. Manufacturers are continuously coming up with new ways to optimize bandwidth and efficiency, such examples being the memory crossbar interface, hidden surface removal algorithms, depth/color/texture compression, and specialized caches for depth, vertexes, primitives, textures, and pixels. And although both sides have been reluctant to give more detailed information about their 3D pipelines, it is most likely that they have many significant architectural differences. Thus, it is becoming more and more difficult to make fair comparisons with a single test, and if this continues to happen then benchmarks will eventually lose all credibility.

The approach Futuremark wants to take is to create separate optimized builds or code paths for both camps (much like game developers), each utilizing its corresponding platform's unique architecture to the fullest extent with standardized non-vendor extensions (ie, OpenGL) and thus allowing for more accurate, reliable comparisons. nVidia and ATi will continue to work closely with Futuremark so that both the benchmark and the hardware will improve. This could potentially be a very good thing.

Of course, there are some very major problems involved, mainly being the influence of money. Business is business, and the company that can give more cash will be able to throw its weight around more easily. The only way that 3DMark will remain a valid benchmark is if they drop the BETA program's fee entirely, removing most fear and paranoia of sneaky corporate tactics (although this will probably never happen).

In addition, there will be endless arguments as to how precise certain functions of the benchmark should be carried out, especially when it can affect performance as much as it does now. The fact is that there is no current solution to this, because there are so many different methods of implementation and the performance range varies so wildly that there can be no "right way" of doing it. Again, the solution will come in time when we have rock solid standards that define a single precision for each type of register in the fragment shader and the manufacturers (ideally) comply.

[ImaginaryFiend]

I really don't understand why so many of you are determined to think Nvidia is evil. They're not "the man." :afro: Remember, they're not much more than a startup. They're less than one one-hundredth the size of Intel. They even have fewer employees than ATI.

Also, think about how much they've done for you. I guarantee that our game playing would not be anywhere near the quality it is without Nvidia, whether or not the card currently in your machine was made by them. They invented register combiners. They invented vertex programs. They invented cube maps in hardware. They are the only company that offers fragment programs good enough to do "cinematic shading". These are huge contributions.

As for ATI, they are just as big a company. They play all the same moves as Nvidia. They haven't contributed as much to the industry, and over the long term and for the near future, their products aren't as good (though they were ahead lately).

Remember that ATI cheated on 3dmark, too. ATI cheats on Quake3. ATI cheats on all trilinear MIPmapping and on aniso filtering.

As for PR fiascos, remember that ATI got in deep trouble with Id for leaking a Doom 3 prerelease last year. They got in big trouble with Apple for leaking specs of Apple's new machines before MacWorld. They got themselves in the Quake/Quack fiasco. They trashed on the Geforce 4 MX's name because it's a DirectX 7 part and stated that ATI would always name their products clearly, with the first digit being its DX version. But then they misled us with the Radeon 9000 and 9200, which are DX 8 parts.

I really think that neither Nvidia nor ATI are evil. They're just trying to make and sell as many of the best products they can, as expected.

Futuremark isn't evil either, but Nvidia's right that their benchmark isn't written the way games are written, making it not very useful for anyone. It makes perfect sense to me that Futuremark and Nvidia would want to resolve this dispute; it's to both their benefit and it doesn't mean that either party caved or that either party strong-armed the other. Disputes are supposed to be resolved. That's progress!

Imaginary Fiend.

[Hellbinder]

ATi did not leak Doom 3. Even Id publcally stated this.

Another case where Nvidia fansites continue to spread misinformation and at the same time try to claim that Nvidia is not evil.. :rolleyes:

I also have *factual* issues with nearly every scentance posted in the above thread..

Example

They invented register combiners. They invented vertex programs. They invented cube maps in hardware. They are the only company that offers fragment programs good enough to do "cinematic shading". These are huge contributions.

None of the above is True at all. And thats just to start.

And you guys wonder why i resort to flaming on occasion...

[CtB]

For all the people talking about Intel optimizing for its architecture and Nvidia doing the same, imagine if your CPU started doing only half the work sent to it...

Of course, most of the people with such arguments probably wouldn't know a CPU from a hard drive if it hit them in the face...

[Zenikase]

Intel can only optimize for general purpose functions, since they themselves have no actual control over how well their CPUs will perform in certain applications (unless they are collaborating with that application's developer in improving software support).

nVidia's case is different because they take advantage of their situation with driver compilers and use it so that it detects an application and modifies the existing code to reduce the workload.

[ImaginaryFiend]

Originally posted by Hellbinder
ATi did not leak Doom 3. Even Id publcally stated this.

Another case where Nvidia fansites continue to spread misinformation and at the same time try to claim that Nvidia is not evil.. :rolleyes:

I also have *factual* issues with nearly every scentance posted in the above thread..

Example
They invented register combiners. They invented vertex programs. They invented cube maps in hardware. They are the only company that offers fragment programs good enough to do "cinematic shading".

None of the above is True at all. And thats just to start.

And you guys wonder why i resort to flaming on occasion...




Sounds like you're right about the Id thing. I'd never heard that.

I don't know where you're getting your information, but here's one of Nvidia's several vertex program patents:

Method, apparatus and article of manufacture for a sequencer in a transform/lighting module capable of processing multiple independent execution threads

Remember, vertex programs first came out in the Geforce 3, 6 months before the Radeon 8500.

Here's Nvidia's register combiners patent. It was first used in the Geforce 256 in August 99. The Radeon didn't come out until May 2000.
Graphics pipeline including combiner stages
Cube environment maps were in this same generation. They are in DX7 because Nvidia put them in hardware.

Here's Nvidia's first pixel shader patent. The reason I think they are the only ones capable of cinematic shading is that ATI can only do 64 instruction-long programs. Nvidia can do at least 1024 instructions. Nvidia also has fp32, which cinematic renderers use. ATI also has severe restrictions on the texture lookups, which is unacceptable for renderman-like shaders.

System, method and article of manufacture for pixel shaders for programmable shading

Sweeping dismissals of detailed arguments just don't work. If you're going to disagree with me, it will only be effective if you provide information to back up your point of view. You can't just say I'm wrong and expect anyone to believe you.