Go Back   nV News Forums > Hardware Forums > Benchmarking And Overclocking

Newegg Daily Deals

Reply
 
Thread Tools
Old 11-03-03, 02:47 PM   #1
StealthHawk
Guest
 
Posts: n/a
Default A Benchmark Optimization Inquiry

There has been much speculation over the last few NVIDIA driver releases that some benchmark specific optimizations have been removed from the drivers. In order to substantiate these claims, I have taken the time to look at thirteen different driver revisions and their performance in three benchmarks which have been known to be targeted at various times. These benchmarks are Futuremark's 3dmark2001, Futuremark's 3dmark03, and Codecult's Codecreatures.

Testing Methodology
Testing methodology is as follows. All tests were taken on a GeForceFX5900Ultra with default settings in each benchmark. This means that the results may or may not reflect the state of drivers for non-GeForceFX cards. These results are only relevant for GeForceFX cards and drivers. A variety of drivers ranging from 43.45 to the newest official driver 52.16 were used. Unwinder's RivaTuner anti-detector patch script was used on drivers that it could be applied on. Please pay heed to the following notes about testing:

Driver 43.45 and 43.51 do not officially support the GeForceFX5900. This means that the drivers may or may not allow the GeForceFX5900 to operate properly. I did not see any visual glitches with either of these drivers, but the drivers may not be optimized properly to take advantage of architectural changes between NV30 and NV35. This means that the results may be invalid. On the other hand, they may also be valid. Since this is somewhat of an unknown, less weight will be given to these two driver revisions. I do feel, however, that it is worth looking at these drivers, because they can help establish a pattern. It should be worth noting that these drivers allow a maximum shader floating point precision of FP16, while every driver officially supporting GeForceFX5900 allows FP32 to be used on a GeForceFX5900.

As mentioned above, anti-detector was used on all drivers where applicable. Anti-detector is used to stop the driver from detecting certain applications. The purpose of this is to see what the true unoptimized performance of a graphics card is in a synthetic benchmark. Such scores are denoted by "[driver] AD." In the real world, not all application detection is synonymous with illegitimate optimization, it may be used for bug fixing, etc. In the realm of synthetic benchmarks however, application detection of major benchmarking programs clearly has only one purpose: to inflate scores artificially.

Codecreatures


Every single driver shows relatively the same score. When anti-detect is used, the score drops off by over 20% no matter which driver is tested. It seems pretty clear that Codecreatures is probably still being optimized for. It can be noted, that this is an old benchmark that is hardly used anymore, so this could be a vestigial optimization that has been overlooked, much like ATI's 3dmark2001 GT4 optimization in Nature which AFAIK has still not been removed either.

3dmark2001


For 3dmark2001, attention is immediately drawn to GT2 Low Detail, GT4, Pixel Shader, and the Advanced Pixel Shader tests due to discrepancies between regular and anti-detect scores.

GT1 Low Detail shows a few FPS variance, but this does not necessarily mean it is being optimized since the difference is usually very minuscule.

GT2 Low Detail always loses performance with anti-detect regardless of what driver is used. This test seems suspect.

GT4 with anti-detect is consistently in the 70s, with results that corroborate themselves. With 43.45, GT4 scores are in the 70s. Driver 43.51 to 44.90 boost the score to ~120fps. Scores fall to the 90s with later drivers, suggesting that some optimizations were removed from the driver, but certainly not all of them.

Pixel Shader scores in the mid to low 200s with driver 43.45 to 44.65. Anti-detect scores are consistently around 190fps. Starting with 44.67, the Pixel Shader test scores fall to 190fps, the same score as shown with anti-detect in earlier drivers. It seems that the optimization here was removed.

The Advanced Pixel Shader test shows differences in the scores when anti-detect is applied up to the 50 series driver, where the anti-detect and regular score is the same. This again suggests that an optimization was in place in earlier drivers, but removed in 51.75.

3dmark2003


The results generally speak for themselves. By driver 44.65 all tests stressing the graphics card are detected and optimized, besides fillrate tests which are not shown. From 44.65 to 44.03 the optimizations actually increased. This is easily explained, however. All 3dmark03 testing done here used the 330 build of 3dmark03. It is well known that the 330 build disabled some detection, although it seems that it didn't catch it all in drivers 43.45, 43.51, and 44.03. 44.65 brought scores back up to the highs seen with the 320 build and older drivers. In essence, 44.65 re-enabled optimizations, or new optimizations were created which resulted in similar scores.

Conclusion
All in all, I am very disappointed in these results. After hearing reports of benchmark scores dropping with several different driver revisions, I was hopeful that benchmark specific optimizations had been removed. It seems that some were removed from 3dmark2001. Codecreatures and 3dmark03 remain just as optimized for as they've always been. It seemed plausible to me that optimizations might have been removed due to Futuremark's ultimatums. Even though two of these benchmarks are a little old, and oft used anymore, it would be nice to see old optimizations removed in good faith.

It will be interesting to see how NVIDIA drivers develop in the future with regards to 3dmark2001 and 3dmark03. On September 23, Futuremark announced their optimization guidelines. Futuremark did say that starting on October 31 the guidelines would start to be enforced. However, the date recently passed and all we got was a statement about enforcing the guidelines, saying that a new build of the popular benchmark(s) will be available in the future. There is still no word on when Futuremark will let us know which drivers are "legal" and optimization free. Once Futuremark gets around to enforcing their rules things might get interesting.
  Reply With Quote
Old 11-03-03, 03:05 PM   #2
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Wow! Nice job StealthHawk, that must have been a mind-numbing benching session to get all them numbers!

Interesting results, I can't wait for FM to release the newest 3dm2k3 patch.
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Old 11-03-03, 03:15 PM   #3
DMA
Zzzleepy
 
DMA's Avatar
 
Join Date: Apr 2003
Location: In bed
Posts: 997
Default

omg..we have to help you guys to find jobs so you stop caring about this
__________________
Gigabyte EP35-DS4 | E6850@3.8GHz |
8800GT@800/2000MHz | 4 GB Corsair 6400|
DMA is offline   Reply With Quote
Old 11-03-03, 03:53 PM   #4
ChrisW
"I was wrong", said Chris
 
Join Date: Jul 2002
Location: standing in the corner!
Posts: 620
Default

We already know nVidia is still doing per-application optimizations as they said they were and would continue to do it. The question is just what are these optimizations and do they go too far?
__________________
AIW 9700 Pro | 1.3GHz CeleronT | 512MB PC133 SDRAM | ECS PCIPAT Mobo (Intel 815EP)
RadLinker/RadClocker
ChrisW is offline   Reply With Quote
Old 11-03-03, 03:59 PM   #5
bkswaney
Mr. Extreme!
 
bkswaney's Avatar
 
Join Date: Aug 2002
Location: SC
Posts: 3,421
Send a message via Yahoo to bkswaney
Default

Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.
Everyone knows that the FX line has to have them or it's a dog.


Thanx for all that hard work. Man that was a lot of benching.
__________________
Notebook!
Compaq Presario CQ60-215DX
AMD 64 Athlon X2 @ 2GHz (QL62)
15.6 inch HD WideScreen
Nvidia 8200M-G 895mb
2Gig system ram
250Gig SATA 5400rpm HDrive
Vista Premium
bkswaney is offline   Reply With Quote
Old 11-03-03, 04:33 PM   #6
Hanners
Elite Bastard
 
Hanners's Avatar
 
Join Date: Jan 2003
Posts: 984
Default

Quote:
Originally posted by bkswaney
Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.
In games, that theory is fine. In synthetic benchmarks, I couldn't disagree more.
__________________
Owner / Editor-in-Chief - Elite Bastards
Hanners is offline   Reply With Quote
Old 11-03-03, 04:53 PM   #7
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Quote:
Originally posted by bkswaney
Yep... I want them to opti as long as my IQ is great
and looks like it suppose to I have no problem with it.
Everyone knows that the FX line has to have them or it's a dog.
In games and applications, yes; but for benchmarks it is just plain wrong and bordering on consumer fraud.

I'm not against any optimizations for games, so long as any IQ reducing ones can be turned on/off and as long as they don't try and hide 'em.
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Old 11-03-03, 05:12 PM   #8
bkswaney
Mr. Extreme!
 
bkswaney's Avatar
 
Join Date: Aug 2002
Location: SC
Posts: 3,421
Send a message via Yahoo to bkswaney
Default

Sorry that is the way I feel to.
I should have said "games"

At least I know now the best drivers for my benchmark scores.
44.03's it is.

Last edited by bkswaney; 11-03-03 at 08:38 PM.
bkswaney is offline   Reply With Quote

Old 11-03-03, 08:35 PM   #9
bloodbob
Registered User
 
Join Date: Oct 2002
Posts: 123
Default

They have to relaese a new version because they need to put out a new ELUA so they can enforce it because otherwise the user is legally allowed to publish results for cheating drivers with their products.
__________________
I come from planet viper days so don't call me noob. I own 2 nvidia cards and one ati card so don't call me biased.
bloodbob is offline   Reply With Quote
Old 11-03-03, 08:43 PM   #10
Ruined
Registered User
 
Ruined's Avatar
 
Join Date: Jul 2003
Posts: 2,447
Default

Quote:
Originally posted by bkswaney
Sorry that is the way I feel to.
I should have said "games"

At least I know now the best drivers for my benchmark scores.
44.03's it is.
Not really, 44.03's don't have the legit compiler optimizations of det50
__________________
We're all in it together.

Intel Core 2 Quad Q6700 2.66GHz CPU | Intel G965WH mobo | 8GB (4x2GB) DDR2-667mhz CAS5 RAM (1066MHz FSB) | BFG GeForce 285 GTX OC 1GB | Dell E228WFP 22" DVI-HDCP LCD Monitor | 1TB Western Digital RE3 SATA2 Main Drive | 500GBx2 Western Digital RE3 SATA2 Scratch Drives in RAID0 | Western Digital RE3 1TB SATA2 Media Drive | External 2TB Western Digital MyBook Backup Drive | Adaptec eSATA 3.0gbps PCI-E interface | Sandisk External 12-in-1 Flash Card Reader | LG GGC-H20L HD DVD/BD reader, DVD writer | LG GGW-H20L HD DVD/BD reader, DVD/BD writer | Microsoft E4000 Ergonomic Keyboard | Logitech Trackman Wheel | Antec P182 ATX Case | Thermaltake ToughPower XT 850w modular PSU | KRK RP-8 Rokit Studio Monitors | Windows Vista Ultimate x64
Ruined is offline   Reply With Quote
Old 11-03-03, 10:47 PM   #11
GJH
Registered User
 
Join Date: Oct 2003
Location: Aussie
Posts: 4
Default

Hey Stealth can you do tests on ati drivers on 3dmark03 and let us no what you find please mate .
GJH is offline   Reply With Quote
Old 11-03-03, 10:49 PM   #12
The Baron
Guest
 
Posts: n/a
Default

Quote:
Originally posted by GJH
Hey Stealth can you do tests on ati drivers on 3dmark03 and let us no what you find please mate .
What's the first driver that supports a 9600 Pro? I could go back and try it before I yoink this card out in a few days (once I have a bit of free time).
  Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Similar Threads
Thread Thread Starter Forum Replies Last Post
(Quick Test) PLA Direct3D 11 Benchmark (GTX680 vs GTX580 vs GTX480) News Archived News Items 0 06-06-12 09:30 AM
Passion Leads Army: DX 11 and GPU PhysX Benchmark News Archived News Items 0 05-28-12 08:00 PM
Anyone benchmark KT266, KT266A, KT333 and KT400? Needa916 CPUs, Motherboards And Memory 7 10-09-02 08:34 AM
3d benchmark se cricket Benchmarking And Overclocking 3 08-30-02 11:02 PM
Network/Lan benchmark wanted. Any Input? Switch Benchmarking And Overclocking 10 07-30-02 09:54 AM

All times are GMT -5. The time now is 10:33 AM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright ©1998 - 2014, nV News.