nV News Forums

 
 

nV News Forums (http://www.nvnews.net/vbulletin/index.php)
-   Benchmarking And Overclocking (http://www.nvnews.net/vbulletin/forumdisplay.php?f=24)
-   -   A Benchmark Optimization Inquiry (http://www.nvnews.net/vbulletin/showthread.php?t=20526)

StealthHawk 11-03-03 02:47 PM

A Benchmark Optimization Inquiry
 
There has been much speculation over the last few NVIDIA driver releases that some benchmark specific optimizations have been removed from the drivers. In order to substantiate these claims, I have taken the time to look at thirteen different driver revisions and their performance in three benchmarks which have been known to be targeted at various times. These benchmarks are Futuremark's 3dmark2001, Futuremark's 3dmark03, and Codecult's Codecreatures.

Testing Methodology
Testing methodology is as follows. All tests were taken on a GeForceFX5900Ultra with default settings in each benchmark. This means that the results may or may not reflect the state of drivers for non-GeForceFX cards. These results are only relevant for GeForceFX cards and drivers. A variety of drivers ranging from 43.45 to the newest official driver 52.16 were used. Unwinder's RivaTuner anti-detector patch script was used on drivers that it could be applied on. Please pay heed to the following notes about testing:

Driver 43.45 and 43.51 do not officially support the GeForceFX5900. This means that the drivers may or may not allow the GeForceFX5900 to operate properly. I did not see any visual glitches with either of these drivers, but the drivers may not be optimized properly to take advantage of architectural changes between NV30 and NV35. This means that the results may be invalid. On the other hand, they may also be valid. Since this is somewhat of an unknown, less weight will be given to these two driver revisions. I do feel, however, that it is worth looking at these drivers, because they can help establish a pattern. It should be worth noting that these drivers allow a maximum shader floating point precision of FP16, while every driver officially supporting GeForceFX5900 allows FP32 to be used on a GeForceFX5900.

As mentioned above, anti-detector was used on all drivers where applicable. Anti-detector is used to stop the driver from detecting certain applications. The purpose of this is to see what the true unoptimized performance of a graphics card is in a synthetic benchmark. Such scores are denoted by "[driver] AD." In the real world, not all application detection is synonymous with illegitimate optimization, it may be used for bug fixing, etc. In the realm of synthetic benchmarks however, application detection of major benchmarking programs clearly has only one purpose: to inflate scores artificially.

Codecreatures
http://www.mrclam.com/~stealth/codecreatures.png

Every single driver shows relatively the same score. When anti-detect is used, the score drops off by over 20% no matter which driver is tested. It seems pretty clear that Codecreatures is probably still being optimized for. It can be noted, that this is an old benchmark that is hardly used anymore, so this could be a vestigial optimization that has been overlooked, much like ATI's 3dmark2001 GT4 optimization in Nature which AFAIK has still not been removed either.

3dmark2001
http://www.mrclam.com/~stealth/3dmark2001.png

For 3dmark2001, attention is immediately drawn to GT2 Low Detail, GT4, Pixel Shader, and the Advanced Pixel Shader tests due to discrepancies between regular and anti-detect scores.

GT1 Low Detail shows a few FPS variance, but this does not necessarily mean it is being optimized since the difference is usually very minuscule.

GT2 Low Detail always loses performance with anti-detect regardless of what driver is used. This test seems suspect.

GT4 with anti-detect is consistently in the 70s, with results that corroborate themselves. With 43.45, GT4 scores are in the 70s. Driver 43.51 to 44.90 boost the score to ~120fps. Scores fall to the 90s with later drivers, suggesting that some optimizations were removed from the driver, but certainly not all of them.

Pixel Shader scores in the mid to low 200s with driver 43.45 to 44.65. Anti-detect scores are consistently around 190fps. Starting with 44.67, the Pixel Shader test scores fall to 190fps, the same score as shown with anti-detect in earlier drivers. It seems that the optimization here was removed.

The Advanced Pixel Shader test shows differences in the scores when anti-detect is applied up to the 50 series driver, where the anti-detect and regular score is the same. This again suggests that an optimization was in place in earlier drivers, but removed in 51.75.

3dmark2003
http://www.mrclam.com/~stealth/3dmark03.png

The results generally speak for themselves. By driver 44.65 all tests stressing the graphics card are detected and optimized, besides fillrate tests which are not shown. From 44.65 to 44.03 the optimizations actually increased. This is easily explained, however. All 3dmark03 testing done here used the 330 build of 3dmark03. It is well known that the 330 build disabled some detection, although it seems that it didn't catch it all in drivers 43.45, 43.51, and 44.03. 44.65 brought scores back up to the highs seen with the 320 build and older drivers. In essence, 44.65 re-enabled optimizations, or new optimizations were created which resulted in similar scores.

Conclusion
All in all, I am very disappointed in these results. After hearing reports of benchmark scores dropping with several different driver revisions, I was hopeful that benchmark specific optimizations had been removed. It seems that some were removed from 3dmark2001. Codecreatures and 3dmark03 remain just as optimized for as they've always been. It seemed plausible to me that optimizations might have been removed due to Futuremark's ultimatums. Even though two of these benchmarks are a little old, and oft used anymore, it would be nice to see old optimizations removed in good faith.

It will be interesting to see how NVIDIA drivers develop in the future with regards to 3dmark2001 and 3dmark03. On September 23, Futuremark announced their optimization guidelines. Futuremark did say that starting on October 31 the guidelines would start to be enforced. However, the date recently passed and all we got was a statement about enforcing the guidelines, saying that a new build of the popular benchmark(s) will be available in the future. There is still no word on when Futuremark will let us know which drivers are "legal" and optimization free. Once Futuremark gets around to enforcing their rules things might get interesting.

digitalwanderer 11-03-03 03:05 PM

Wow! Nice job StealthHawk, that must have been a mind-numbing benching session to get all them numbers! :eek:

Interesting results, I can't wait for FM to release the newest 3dm2k3 patch.

DMA 11-03-03 03:15 PM

omg..we have to help you guys to find jobs so you stop caring about this :D

ChrisW 11-03-03 03:53 PM

We already know nVidia is still doing per-application optimizations as they said they were and would continue to do it. The question is just what are these optimizations and do they go too far?

bkswaney 11-03-03 03:59 PM

Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.
Everyone knows that the FX line has to have them or it's a dog.


Thanx for all that hard work. Man that was a lot of benching. :eek:

Hanners 11-03-03 04:33 PM

Quote:

Originally posted by bkswaney
Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.

In games, that theory is fine. In synthetic benchmarks, I couldn't disagree more.

digitalwanderer 11-03-03 04:53 PM

Quote:

Originally posted by bkswaney
Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.
Everyone knows that the FX line has to have them or it's a dog.

In games and applications, yes; but for benchmarks it is just plain wrong and bordering on consumer fraud.

I'm not against any optimizations for games, so long as any IQ reducing ones can be turned on/off and as long as they don't try and hide 'em.

bkswaney 11-03-03 05:12 PM

Sorry that is the way I feel to.
I should have said "games" :)

At least I know now the best drivers for my benchmark scores.
44.03's it is. ;)

bloodbob 11-03-03 08:35 PM

They have to relaese a new version because they need to put out a new ELUA so they can enforce it because otherwise the user is legally allowed to publish results for cheating drivers with their products.

Ruined 11-03-03 08:43 PM

Quote:

Originally posted by bkswaney
Sorry that is the way I feel to.
I should have said "games" :)

At least I know now the best drivers for my benchmark scores.
44.03's it is. ;)

Not really, 44.03's don't have the legit compiler optimizations of det50

GJH 11-03-03 10:47 PM

Hey Stealth can you do tests on ati drivers on 3dmark03 and let us no what you find please mate .

The Baron 11-03-03 10:49 PM

Quote:

Originally posted by GJH
Hey Stealth can you do tests on ati drivers on 3dmark03 and let us no what you find please mate .
What's the first driver that supports a 9600 Pro? I could go back and try it before I yoink this card out in a few days (once I have a bit of free time).


All times are GMT -5. The time now is 09:36 AM.

Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.