StealthHawk
11-03-03, 04:47 PM
There has been much speculation over the last few NVIDIA driver releases that some benchmark specific optimizations have been removed from the drivers. In order to substantiate these claims, I have taken the time to look at thirteen different driver revisions and their performance in three benchmarks which have been known to be targeted at various times. These benchmarks are Futuremark's 3dmark2001, Futuremark's 3dmark03, and Codecult's Codecreatures.
Testing Methodology
Testing methodology is as follows. All tests were taken on a GeForceFX5900Ultra with default settings in each benchmark. This means that the results may or may not reflect the state of drivers for non-GeForceFX cards. These results are only relevant for GeForceFX cards and drivers. A variety of drivers ranging from 43.45 to the newest official driver 52.16 were used. Unwinder's RivaTuner anti-detector patch script was used on drivers that it could be applied on. Please pay heed to the following notes about testing:
Driver 43.45 and 43.51 do not officially support the GeForceFX5900. This means that the drivers may or may not allow the GeForceFX5900 to operate properly. I did not see any visual glitches with either of these drivers, but the drivers may not be optimized properly to take advantage of architectural changes between NV30 and NV35. This means that the results may be invalid. On the other hand, they may also be valid. Since this is somewhat of an unknown, less weight will be given to these two driver revisions. I do feel, however, that it is worth looking at these drivers, because they can help establish a pattern. It should be worth noting that these drivers allow a maximum shader floating point precision of FP16, while every driver officially supporting GeForceFX5900 allows FP32 to be used on a GeForceFX5900.
As mentioned above, anti-detector was used on all drivers where applicable. Anti-detector is used to stop the driver from detecting certain applications. The purpose of this is to see what the true unoptimized performance of a graphics card is in a synthetic benchmark. Such scores are denoted by "[driver] AD." In the real world, not all application detection is synonymous with illegitimate optimization, it may be used for bug fixing, etc. In the realm of synthetic benchmarks however, application detection of major benchmarking programs clearly has only one purpose: to inflate scores artificially.
Codecreatures
http://www.mrclam.com/~stealth/codecreatures.png
Every single driver shows relatively the same score. When anti-detect is used, the score drops off by over 20% no matter which driver is tested. It seems pretty clear that Codecreatures is probably still being optimized for. It can be noted, that this is an old benchmark that is hardly used anymore, so this could be a vestigial optimization that has been overlooked, much like ATI's 3dmark2001 GT4 optimization in Nature which AFAIK has still not been removed either.
3dmark2001
http://www.mrclam.com/~stealth/3dmark2001.png
For 3dmark2001, attention is immediately drawn to GT2 Low Detail, GT4, Pixel Shader, and the Advanced Pixel Shader tests due to discrepancies between regular and anti-detect scores.
GT1 Low Detail shows a few FPS variance, but this does not necessarily mean it is being optimized since the difference is usually very minuscule.
GT2 Low Detail always loses performance with anti-detect regardless of what driver is used. This test seems suspect.
GT4 with anti-detect is consistently in the 70s, with results that corroborate themselves. With 43.45, GT4 scores are in the 70s. Driver 43.51 to 44.90 boost the score to ~120fps. Scores fall to the 90s with later drivers, suggesting that some optimizations were removed from the driver, but certainly not all of them.
Pixel Shader scores in the mid to low 200s with driver 43.45 to 44.65. Anti-detect scores are consistently around 190fps. Starting with 44.67, the Pixel Shader test scores fall to 190fps, the same score as shown with anti-detect in earlier drivers. It seems that the optimization here was removed.
The Advanced Pixel Shader test shows differences in the scores when anti-detect is applied up to the 50 series driver, where the anti-detect and regular score is the same. This again suggests that an optimization was in place in earlier drivers, but removed in 51.75.
3dmark2003
http://www.mrclam.com/~stealth/3dmark03.png
The results generally speak for themselves. By driver 44.65 all tests stressing the graphics card are detected and optimized, besides fillrate tests which are not shown. From 44.65 to 44.03 the optimizations actually increased. This is easily explained, however. All 3dmark03 testing done here used the 330 build of 3dmark03. It is well known that the 330 build disabled some detection, although it seems that it didn't catch it all in drivers 43.45, 43.51, and 44.03. 44.65 brought scores back up to the highs seen with the 320 build and older drivers. In essence, 44.65 re-enabled optimizations, or new optimizations were created which resulted in similar scores.
Conclusion
All in all, I am very disappointed in these results. After hearing reports of benchmark scores dropping with several different driver revisions, I was hopeful that benchmark specific optimizations had been removed. It seems that some were removed from 3dmark2001. Codecreatures and 3dmark03 remain just as optimized for as they've always been. It seemed plausible to me that optimizations might have been removed due to Futuremark's ultimatums. Even though two of these benchmarks are a little old, and oft used anymore, it would be nice to see old optimizations removed in good faith.
It will be interesting to see how NVIDIA drivers develop in the future with regards to 3dmark2001 and 3dmark03. On September 23, Futuremark announced their optimization guidelines (http://www.futuremark.com/pressroom/pressreleases/?092303). Futuremark did say that starting on October 31 the guidelines would start to be enforced. However, the date recently passed and all we got was a statement about enforcing the guidelines (http://www.futuremark.com/companyinfo/Enforcement_Process.pdf), saying that a new build of the popular benchmark(s) will be available in the future. There is still no word on when Futuremark will let us know which drivers are "legal" and optimization free. Once Futuremark gets around to enforcing their rules things might get interesting.
Testing Methodology
Testing methodology is as follows. All tests were taken on a GeForceFX5900Ultra with default settings in each benchmark. This means that the results may or may not reflect the state of drivers for non-GeForceFX cards. These results are only relevant for GeForceFX cards and drivers. A variety of drivers ranging from 43.45 to the newest official driver 52.16 were used. Unwinder's RivaTuner anti-detector patch script was used on drivers that it could be applied on. Please pay heed to the following notes about testing:
Driver 43.45 and 43.51 do not officially support the GeForceFX5900. This means that the drivers may or may not allow the GeForceFX5900 to operate properly. I did not see any visual glitches with either of these drivers, but the drivers may not be optimized properly to take advantage of architectural changes between NV30 and NV35. This means that the results may be invalid. On the other hand, they may also be valid. Since this is somewhat of an unknown, less weight will be given to these two driver revisions. I do feel, however, that it is worth looking at these drivers, because they can help establish a pattern. It should be worth noting that these drivers allow a maximum shader floating point precision of FP16, while every driver officially supporting GeForceFX5900 allows FP32 to be used on a GeForceFX5900.
As mentioned above, anti-detector was used on all drivers where applicable. Anti-detector is used to stop the driver from detecting certain applications. The purpose of this is to see what the true unoptimized performance of a graphics card is in a synthetic benchmark. Such scores are denoted by "[driver] AD." In the real world, not all application detection is synonymous with illegitimate optimization, it may be used for bug fixing, etc. In the realm of synthetic benchmarks however, application detection of major benchmarking programs clearly has only one purpose: to inflate scores artificially.
Codecreatures
http://www.mrclam.com/~stealth/codecreatures.png
Every single driver shows relatively the same score. When anti-detect is used, the score drops off by over 20% no matter which driver is tested. It seems pretty clear that Codecreatures is probably still being optimized for. It can be noted, that this is an old benchmark that is hardly used anymore, so this could be a vestigial optimization that has been overlooked, much like ATI's 3dmark2001 GT4 optimization in Nature which AFAIK has still not been removed either.
3dmark2001
http://www.mrclam.com/~stealth/3dmark2001.png
For 3dmark2001, attention is immediately drawn to GT2 Low Detail, GT4, Pixel Shader, and the Advanced Pixel Shader tests due to discrepancies between regular and anti-detect scores.
GT1 Low Detail shows a few FPS variance, but this does not necessarily mean it is being optimized since the difference is usually very minuscule.
GT2 Low Detail always loses performance with anti-detect regardless of what driver is used. This test seems suspect.
GT4 with anti-detect is consistently in the 70s, with results that corroborate themselves. With 43.45, GT4 scores are in the 70s. Driver 43.51 to 44.90 boost the score to ~120fps. Scores fall to the 90s with later drivers, suggesting that some optimizations were removed from the driver, but certainly not all of them.
Pixel Shader scores in the mid to low 200s with driver 43.45 to 44.65. Anti-detect scores are consistently around 190fps. Starting with 44.67, the Pixel Shader test scores fall to 190fps, the same score as shown with anti-detect in earlier drivers. It seems that the optimization here was removed.
The Advanced Pixel Shader test shows differences in the scores when anti-detect is applied up to the 50 series driver, where the anti-detect and regular score is the same. This again suggests that an optimization was in place in earlier drivers, but removed in 51.75.
3dmark2003
http://www.mrclam.com/~stealth/3dmark03.png
The results generally speak for themselves. By driver 44.65 all tests stressing the graphics card are detected and optimized, besides fillrate tests which are not shown. From 44.65 to 44.03 the optimizations actually increased. This is easily explained, however. All 3dmark03 testing done here used the 330 build of 3dmark03. It is well known that the 330 build disabled some detection, although it seems that it didn't catch it all in drivers 43.45, 43.51, and 44.03. 44.65 brought scores back up to the highs seen with the 320 build and older drivers. In essence, 44.65 re-enabled optimizations, or new optimizations were created which resulted in similar scores.
Conclusion
All in all, I am very disappointed in these results. After hearing reports of benchmark scores dropping with several different driver revisions, I was hopeful that benchmark specific optimizations had been removed. It seems that some were removed from 3dmark2001. Codecreatures and 3dmark03 remain just as optimized for as they've always been. It seemed plausible to me that optimizations might have been removed due to Futuremark's ultimatums. Even though two of these benchmarks are a little old, and oft used anymore, it would be nice to see old optimizations removed in good faith.
It will be interesting to see how NVIDIA drivers develop in the future with regards to 3dmark2001 and 3dmark03. On September 23, Futuremark announced their optimization guidelines (http://www.futuremark.com/pressroom/pressreleases/?092303). Futuremark did say that starting on October 31 the guidelines would start to be enforced. However, the date recently passed and all we got was a statement about enforcing the guidelines (http://www.futuremark.com/companyinfo/Enforcement_Process.pdf), saying that a new build of the popular benchmark(s) will be available in the future. There is still no word on when Futuremark will let us know which drivers are "legal" and optimization free. Once Futuremark gets around to enforcing their rules things might get interesting.