My thoughts on "Optimization"

[jimmyjames123]

I have been reading about this 3dmark03 "Rendering Issue".

For one, I think the journalism at Extremetech and Beyond3d was a bit odd. Never do they mention the following relevant points:

1) Image quality was NOT compromised with the new FX 44.03 driver. In fact, most people have noted improvements in Image Quality, even in 3dmark03.

2) All images in 3dmark03 that we actually SEE are rendered correctly.

These points might sound obvious to some, but to someone casually reading or to someone who isn't very familiar with graphics card terminology, these points wouldn't be so obvious.

Now, ask yourself this question:

If we can get smoother and faster performance, without any loss of image quality, isn't that a good thing?

If I had an FX NVIDIA card, I would be happy that NVIDIA could "optimize" for 3dmark03 without any loss of image quality and without affecting any images that we actually see on screen

It is no secret that both NVIDIA and ATI "optimize" their drivers for a benchmark like 3dmark03. If they can do it without compromising image quality, all the better (IMO).

Some other points that are worth noting:

1) NVIDIA does not have authorized access to the developer's build of 3dmark, while ATI (and Extremetech, Beyond3d, etc) does, where they can roam around anywhere even off-center of the actual image displayed.

2) Futuremark strangely only allows WHQL certified drivers for published online results for their 3dmark program, but they allow overclocked graphics cards and cpu's (and note how Futuremark buries the Detonator FX driver underneath the others on their homepage annoucements, not even mentioning about how it is WHQL certified for GeForce FX cards).

3) [H]OCP talked with ATI privately about the quake/quack driver cheat issue for more than a month before writing their article. ATI repeatedly denied cheating, and [H]OCP released their article after they proved them wrong (showing that image quality was compromised on Quake 3 while enhancing performance).

4) The NVIDIA FX Detonator driver increased both performance and IQ noticeably for a variety of benchmarks and games on the FX graphics cards, not simply 3dmark03.

5) The Doom3 benchmark's used in the latest tests were chosen by ID and were not seen by NVIDIA in advance.

All in all, it boils down to what one considers an "optimization" vs. a "cheat". If a graphics card company can make performance smoother and faster without compromising image quality and without affecting what we actually see, then I would say this is a good thing (especially considering that NVIDIA and ATI both optimize for benchmarks in the first place).

[Dazz]

LMAO who the hells gonna admit they are cheating LMFAO

[digitalwanderer]

Originally posted by jimmyjames123
1) Image quality was NOT compromised with the new FX 44.03 driver. In fact, most people have noted improvements in Image Quality, even in 3dmark03.

2) All images in 3dmark03 that we actually SEE are rendered correctly.

Well, that is definately a nice summation of one popular point of view around here. There seems to be just about as many people though who'd jump up and say something like, "But it is a BENCHMARK, and it is meant to be run in a certain way so as to give an accurate comparison of performance. nVidia's driver cheats compromise that by bypassing the benchmark's parameters, which is NOT optimizing but is fraudulently inflating their scores to sell more cards."

I'm one of them people. :)

Now if this were a GAME we were talking about I would agree with you, as optimizations that don't affect image quality/what you see ARE pure optimizations and nothing but a benefit to the consumer....but this ain't a game and the cheating is cheeseball. ;)

[reever2]

Why dont people understand that this optimization can only be done in a benchmark? Static clip planes can not be put into games, if at any time the camera in 3dmark turned around you would see the clipping, i would say not being able to see 3/4 of the environment a huge detrimental effect to IQ

[jimmyjames123]

"But it is a BENCHMARK, and it is meant to be run in a certain way so as to give an accurate comparison of performance. nVidia's driver cheats compromise that by bypassing the benchmark's parameters, which is NOT optimizing but is fraudulently inflating their scores to sell more cards."

Trust me, I fully understand this point of view. However, we all know that NVIDIA and ATI actively "optimize" their drivers for enhanced performance in 3dmark programs. This alone undercuts the argument about an "accurate comparision of performance", because it has been repeatedly shown that driver "optimizations" alone can (sometimes significantly) improve performance.

There is also the issue about what is an "optimization" and what is a "cheat". There is no agreement about what this distinction is. If NVIDIA (or ATI for that matter) can improve performance without compromising image quality, I'd like to think of that as an "optimization".

If anything, maybe this will help Futuremark to create a benchmark that is less susceptible to "optimizations", assuming that they truly want to create an "impartial" benchmarking program.

[jimmyjames123]

if at any time the camera in 3dmark turned around you would see the clipping, i would say not being able to see 3/4 of the environment a huge detrimental effect to IQ

The fact of the matter is that the cameras in 3dmark don't get turned around. So NVIDIA is "optimizing" for this particular benchmark. It is certainly no secret that both NVIDIA and ATI optimize for this benchmark. Users of FX cards now get smoother and faster performance and better image quality in 3dmark03 in what we can see. This whole issue is a matter of perspective.

[dpollard55]

Originally posted by jimmyjames123
The fact of the matter is that the cameras in 3dmark don't get turned around. So NVIDIA is "optimizing" for this particular benchmark. It is certainly no secret that both NVIDIA and ATI optimize for this benchmark. Users of FX cards now get smoother and faster performance and better image quality in 3dmark03 in what we can see. This whole issue is a matter of perspective.
Well said, and my thoughts exactly.

[digitalwanderer]

Originally posted by jimmyjames123
Trust me, I fully understand this point of view. However, we all know that NVIDIA and ATI actively "optimize" their drivers for enhanced performance in 3dmark programs. This alone undercuts the argument about an "accurate comparision of performance", because it has been repeatedly shown that driver "optimizations" alone can (sometimes significantly) improve performance.

There is also the issue about what is an "optimization" and what is a "cheat". There is no agreement about what this distinction is. If NVIDIA (or ATI for that matter) can improve performance without compromising image quality, I'd like to think of that as an "optimization".

If anything, maybe this will help Futuremark to create a benchmark that is less susceptible to "optimizations", assuming that they truly want to create an "impartial" benchmarking program.

Just because nVidia got busted with a flat-out cheat it does NOT invalidate 3dm2k3 as a benchmark, it invalidates nVidia as a trusted manufacturer. :(

[jimmyjames123]

Just because nVidia got busted with a flat-out cheat it does NOT invalidate 3dm2k3 as a benchmark, it invalidates nVidia as a trusted manufacturer.

According to that logic, ATI should not have been a trusted manufacturer ever since the quake/quack fiasco. As for 3dmark03 as a benchmark, I think there has already been enough talk about the shortcomings of this program as an "accurate" real-world benchmark tool. Also, we are back at the issue of "optimization" vs. "cheat".

I don't know how many times this has to be repeated, but here we go again: It is certainly no secret that both NVIDIA and ATI optimize for this benchmark. Users of FX cards now get smoother and faster performance and better image quality in 3dmark03 in what we can see.

[John Reynolds]

Originally posted by jimmyjames123
I don't know how many times this has to be repeated, but here we go again: It is certainly no secret that both NVIDIA and ATI optimize for this benchmark. Users of FX cards now get smoother and faster performance and better image quality in 3dmark03 in what we can see.

Someone started a thread on this forum before the ET article was published stating that they were seeing pixel smearing in 3DMark.

Plain and simple, whether you want to call it a cheat or an optimization, Nvidia boards are not rendering this synthetic correctly. The human eye might have a hard time detecting this, but it's there nonetheless.

[jimmyjames123]

Someone started a thread on this forum before the ET article was published stating that they were seeing pixel smearing in 3DMark. Plain and simple, whether you want to call it a cheat or an optimization, Nvidia boards are not rendering this synthetic correctly. The human eye might have a hard time detecting this, but it's there nonetheless.

I've read several reviews on the FX 5900 Ultra, and no reviewer has mentioned any corruption of image quality with 3dmark03 using the new Detonator FX 44.03 drivers with what we can actually see. In fact, it seems that both Image quality and performance are improved, sometimes significantly. Also, I read somewhere about how there was a pixel smear due to a video capture error, not an smearing of the image in the actual demo.

[Dazz]

Originally posted by jimmyjames123
I've read several reviews on the FX 5900 Ultra, and no reviewer has mentioned any corruption of image quality with 3dmark03 using the new Detonator FX 44.03 drivers. In fact, it seems that both Image quality and performance are improved, sometimes significantly. Thats due to people leaving the bench running while they are away from the computer.

[digitalwanderer]

Originally posted by jimmyjames123
According to that logic, ATI should not have been a trusted manufacturer ever since the quake/quack fiasco. [/B]

ATi WASN'T a trusted manufacturerer for a looong time after that, at least in the ATi community that I recall. That's why I always refer to 'em as either the "old ATi" or the "spiffy new ATi".

Seriously. They took their lumps for being busted, and they had to work hard to regain the communities respect/trust..but they did and now they're reaping the rewards of that as is the ATi community. (IMHO, and all that rot. ;) )

nVidia needs to take a good hard look at their current business practices and start to realize that the public does NOT like being decieved and a company getting caught with their hand in the cookie jar TWICE in a short period of time after an over-hyped and much delayed launch should probably take some time to reflect over their policies and practices before they damage their reputation beyond repair. :(

I'm serious, and I'm not trying to flame. Even if you don't agree with my position on 3dm2k3 can you see how this could be very bad for nVidia in the long run? I mean, there ARE quite a few people besides me who I feel are thinking this way...and they ain't all just ATi fanboys. ;)

[gordon151]

Originally posted by jimmyjames123
The fact of the matter is that the cameras in 3dmark don't get turned around. So NVIDIA is "optimizing" for this particular benchmark. It is certainly no secret that both NVIDIA and ATI optimize for this benchmark. Users of FX cards now get smoother and faster performance and better image quality in 3dmark03 in what we can see. This whole issue is a matter of perspective.

I don't think this can specifically be called "optimizing" since you are essentially altering the benchmark itself. I can see with optimizing the drivers to improve execution of a specific coding routine that is employed by the benchmark (which from my understanding is the common and preferred practice), but effectively ommitting part of the benchmark from rendering changes the rules as to how the benchmark is run.

This completely skews the scores in a way that driver optimizations wouldn't (as coding for a specific routine is a universal rather than specific application of optimizing) and it would make sense if this were applied to all other cards, that their scores would normalize to where they were before.

[jimmyjames123]

ATi WASN'T a trusted manufacturerer for a looong time after that, at least in the ATi community that I recall. That's why I always refer to 'em as either the "old ATi" or the "spiffy new ATi".
Seriously. They took their lumps for being busted, and they had to work hard to regain the communities respect/trust..but they did and now they're reaping the rewards of that as is the ATi community.

I think you have to look at the fundamental difference between these two situations: ATI's quake/quack issue involved degradation of actual image quality that you could see in the game, for the sake of higher performance. This current 3dmark03 issue involves something that almost no one can see (unless you pay Futuremark for their developer version and roam around off camera), and it involves no degradation of image quality in what we can actually see. Of course, we are also comparing a benchmark that many people feel is not representative of real-world gaming performance (3dmark03) to a very popular game (Quake 3).

[jimmyjames123]

I can see with optimizing the drivers to improve execution of a specific coding routine that is employed by the benchmark (which from my understanding is the common and preferred practice), but effectively ommitting part of the benchmark from rendering changes the rules as to how the benchmark is run.

The idea is that there are no concrete "rules as to how the benchmark is run". There is no concrete definition of what makes up a driver "optimization". NVIDIA does not even have authorized access to the 3dmark03 developer tools that some other websites are using. Futuremark themselves is very inconsistent. They don't allow non-WHQL drivers, but they allow overclocked graphics cards and cpu's. Don't you think that "skews" results in a way that driver "optimizations" wouldn't? That introduces yet another variable, and the graphics cards (and cpu's) that overclock better will have a natural advantage.

[nVidi0t]

It's pretty obvious nVidia cheated.. but.

I have no problem with clip planes if I cant see them :)

[digitalwanderer]

Originally posted by jimmyjames123
The idea is that there are no concrete "rules as to how the benchmark is run". There is no concrete definition of what makes up a driver "optimization". NVIDIA does not even have authorized access to the 3dmark03 developer tools that some other websites are using. Futuremark themselves is very inconsistent. They don't allow non-WHQL drivers, but they allow overclocked graphics cards and cpu's. Don't you think that "skews" results in a way that driver "optimizations" wouldn't? That introduces yet another variable, and the graphics cards (and cpu's) that overclock better will have a natural advantage.

<sigh>

Can we just agree to disagree on this one? I think that's what the crux of the whole debate is, and the peeps who think it's a cheat to "optimize" a benchmark the way nVidia did aren't going to change their minds anymore than the people who believe it's a legitimate "optimization"

Methinks we'll find out more tomorrow, further debating this point is just an exercise in futility right now.

[jjjayb]

This was posted by Joe Defuria over at beyond3d. I think it pretty much sums up why this is a cheat and not an optimization even though image quality stays the same so I'll just paste that quote here:



If the basis of your optimization requires you to have access to data that is NOT PASSED by the game engine in real time, then that optimization is a cheat. This 3DMark cheat is based on the fact that the drivers "are told" the camera path won't change from some determined path. Problem is, they are not told this by the game engine. Clipping planes are inserted based on this knowledge. That data (the clipping planes) are not passed from the engine in real-time, nor are those planes calculated in real-time (as evidenced by the lack of correct rendering when "off the rail".)
That is why this particular example is a cheat, and not a legal optimization. It relies on data that is not given by the benchmark, or calculated in real-time from data given by the benchmark.

This is why something like a "deferred renderer" is NOT cheating. It's not drawing "everything" either. But it calculates, on the fly, frame by frame, what is needed to be drawn. If you took a deferred renderer "off the rail" it would not suffer the clipping issues.


I couldn't sum it up any better, so I won't even try.

[jimmyjames123]

I have no problem with clip planes if I cant see them

Exactly. And as has been noted earlier, image quality and performance with the new Detonator FX 44.03 drivers is actually improved over previous drivers for the FX graphics cards, for a wide variety of benchmarks and games. Also of note is that NVIDIA's highest "Quality" mode now takes much less of a performance hit than before on the FX cards. All in all, much more to celebrate about than to complain about for GeForce FX owners.

[John Reynolds]

Originally posted by jimmyjames123
Exactly. And as has been noted earlier, image quality and performance with the new Detonator FX 44.03 drivers is actually improved over previous drivers for the FX graphics cards, for a wide variety of benchmarks and games. Also of note is that NVIDIA's highest "Quality" mode now takes much less of a performance hit than before on the FX cards. All in all, much more to celebrate about than to complain about for GeForce FX owners.

No offense, but with logic like that you are a PR department's wet dream.

[digitalwanderer]

Originally posted by John Reynolds
No offense, but with logic like that you are a PR department's wet dream.

Actually, now that you mention 'PR'...methinks me smells a bit of a rat. Mebbe he ain't a PR departments wet dream, mebbe he's a PR departments plant. :mad:

You wouldn't happen to work for nVidia in any way, shape, or capacity jimmyjames123...would ya? I mean you've got 8 posts here to date and all you've done is start a thread to try and minimize nVidia's complicity/wrong-doing. :eek2:

And "no", I sure wouldn't put it past nVidia right now. Could a mod check his IP to see if he's silly enough to be posting from nVidia itself? :confused:

[jimmyjames123]

This was posted by Joe Defuria over at beyond3d. I think it pretty much sums up why this is a cheat and not an optimization even though image quality stays the same so I'll just paste that quote here:

Referring to this as an "optimization" or a "cheat" is all ultimately semantics. Even Futuremark doesn't seem to be consisent about how to "accurately" run the 3dmark program (read above about how they allow overclocked graphics cards and cpu's, but not non-WHQL drivers). Also, NVIDIA doesn't have authorized access to the developer's version of 3dmark03 while ATI does. All of this practically throws normalization of the benchmark out the window.

[jimmyjames123]

No offense, but with logic like that you are a PR department's wet dream.

This is how people respond when they have no more relevant points to make :D

[The Baron]

Say what you will--there is no "correct" way to run 3DMark. It's going to be diferent every time. If there was a way to run 3DMark where you could get exactly the same result EVERY TIME you ran it, I would think differently. But there's not. The days of benchmarks as we know them are numbered, but I'll prove that to you in a week or two... ;)