View Full Version : New [H] Editorial on Benchmarking
The Baron
11-13-03, 11:37 AM
Yes, here we go again, ladies. (http://www.hardocp.com/article.html?art=NTQ5)
Reading it now. Should be fun.
John Reynolds
11-13-03, 11:40 AM
Heh, just e-mailed Kyle after reading his editorial. I didn't flame him, but I did point out the amount of PR crap that was being blown up his backside.
The Baron
11-13-03, 11:46 AM
okay. Kyle is insane. I'll leave it at that. he's just nuts.
Actually, I kind of agree with him.
Not all the crap he spewed about nVidia, but some of what he spewed about futuremark.
While I applaud and defend Futuremark for enforcing their rules, and also while I've pointed out that 3DMark is not intended to reflect "real game performance"...
I still think it's a BAD tool.
ANY company that is going to be "responsible" for essentially "policing an industry, via measuring performance", and then takes money from the IHV's and OEM's... well... to me that's a conflict of interest. It open's a whole "can of worms". Who's to say one side isn't "paying more money" than the other side, which is swaying the results? Heck, there's already someone on B3D forums telling Dave that "Dell asked for this patch because they're selling ATi cards, now."
While Futuremark takes money from IHV's and OEM's... I can't really see myself puting any real faith in their product, in general.
Taz
The Baron
11-13-03, 12:10 PM
3DMark03's relevance is dead now. It was not dead when it came out. We have DX9 games now. We didn't have DX9 games then. So... I dunno what I'm trying to say. But it's probably something.
Originally posted by John Reynolds
Heh, just e-mailed Kyle after reading his editorial. I didn't flame him, but I did point out the amount of PR crap that was being blown up his backside.
:rofl
that reminded me of this one guy who went to a gas station and took an air pump and stuck it up his backside.. and exploded lol...
:rofl
John Reynolds
11-13-03, 12:33 PM
Originally posted by The Baron
3DMark03's relevance is dead now. It was not dead when it came out. We have DX9 games now. We didn't have DX9 games then. So... I dunno what I'm trying to say. But it's probably something.
Are you sure about that, because how many game developers continue to police their code in ongoing efforts to circumvent cheating after their game has been published?
I think this quote should explain everything:
This latest patch from Futuremark is yet another revision of 3DMark03 specifically designed to defeat our Unified Compilier Technology, which evaluates shaders and in some cases substitutes hand tuned shaders, but increasingly simply applies the run-time compiler to generate optimal code. With the 52.16 drivers and the new patch, our perf drops 15%.
Clearly our compiler has gotten much better, as image quality remains exactly the same, the only thing that happens is a 10-15% drop in performance.
Notice they used "yet another revision...designed to defeat our Unified Compiler Technology"! The only way for them to be doing this again is for nVidia to just rename all their previous "optimizations" (cheats) to a "Unified Compiler". I think this is official confirmation that is what they are doing. :(
"Substitute hand tuned shaders"? That is exactly what they were doing in the past. People were lead to believe this new "compiler" just re-odered shader instructions to make them much more efficient, not simply detecting shaders and replacing them with new shaders designed to simply achieve a higher score. NVidia should be ashamed of themselves for purposely misleading the public (again). They are smart enough to know that "Unified Compiler Technology" sounds much better than "cheater".
Originally posted by ChrisW
I think this quote should explain everything:
Notice they used "yet another revision...designed to defeat our Unified Compiler Technology"! The only way for them to be doing this again is for nVidia to just rename all their previous "optimizations" (cheats) to a "Unified Compiler". I think this is official confirmation that is what they are doing. :(
AAARGH!:rolleyes:
The compiler is for real guys. Stop inveting BS - everyone is, and it's just annoying me.
The effects of the compiler cannot be as good as hand-tuned code most of the time, but it's pretty darn good. And developers will have making better than what the compiler does.
Disabling the compiler seems like BS though. It's still on, or their scores would be even significantly lower.
Uttar
Originally posted by Uttar
AAARGH!:rolleyes:
The compiler is for real guys. Stop inveting BS - everyone is, and it's just annoying me.
What are you talking about? :confused: That quote came directly from the link. If anyone is making stuff up, it is either nVidia or [H]. It says in the actual quote that they are detecting shaders and replacing them with hand written shaders. I'm not making up anything. That is not my definition of a compiler.
What we expect will happen is that we'll be forced to expend more engineering effort to update our compiler's fingerprinter to be more intelligent, specifically to make it intelligent in its ability to optimize code even when application developers are trying to specifically defeat compilation and optimal code generation.
I don't know about you, but telling us they are going to make it harder for developers to detect when nVidia's drivers are cheating adds a bad taste to my mouth. I don't see how anyone can defend these tactics. And "fingerprinter"...if that is not designed to specifically detect things in the benchmark I don't know what it is. Basically, they are saying they should have made it harder for them to find. I wonder how much other stuff got through they could not find?
Originally posted by ChrisW
I don't know about you, but telling us they are going to make it harder for developers to detect when nVidia's drivers are cheating adds a bad taste to my mouth. I don't see how anyone can defend these tactics. And "fingerprinter"...if that is not designed to specifically detect things in the benchmark I don't know what it is.
Well, their compiler is still running probably. I doubt that FM can disable that. So don't take NV's PR machine to seriously.
ATI's compiler stuff added in one of their driver sets (3.5/3.6?) is probably still alive and kicking too :)
(Dunno if that's the same thing so don't kill me please :D )
Originally posted by The Baron
okay. Kyle is insane. I'll leave it at that. he's just nuts.
Isn't that a bit old news? Seems [H]e has no clue what happens around him, or he doesn't just care. I think we gain nothing by blaming the poor old man for his ignorance... :p
Originally posted by DMA
ATI's compiler stuff added in one of their driver sets (3.5/3.6?) is probably still alive and kicking too :)
(Dunno if that's the same thing so don't kill me please :D )
ATI compiler? I don't know anything about it but I want to know what that is too. If ATI is doing the same thing and calling it a "compiler" then they should be equally chastised.
vampireuk
11-13-03, 01:15 PM
Originally posted by The Baron
okay. Kyle is insane. I'll leave it at that. he's just nuts.
Originally posted by TheTaz
It open's a whole "can of worms". Who's to say one side isn't "paying more money" than the other side, which is swaying the results? Heck, there's already someone on B3D forums telling Dave that "Dell asked for this patch because they're selling ATi cards, now."
Uhhuh, if you've been reading B3D, you should know the relevant changes in the pixel shaders of the new patch are register permutations. If the nVidia unified compiler would be a real deal, it should have no effect on the performance. Actually, the compiler is a real deal, because performance only drops ~15%. It would be a lot more if the compiler would be the old one - the 15 % difference emerges from hand-compiled shaders replacement (cheating, slightly different output).
Furthermore, someone at B3D extracted the old 3dMark patch 330 shaders, and tried them with his own program. Then he tried the patch 340 shaders with his own program. No performance difference - so nVidia drivers are not only doing shader detection but also application detection to make sure only 3dMark03 shaders get replaced. This is obviously not very nice. :mad:
EDIT: Some clarifications...
jbirney
11-13-03, 01:27 PM
Originally posted by The Baron
3DMark03's relevance is dead now. It was not dead when it came out. We have DX9 games now. We didn't have DX9 games then. So... I dunno what I'm trying to say. But it's probably something.
What? Ok so please give me some indications on how HL2 will run. What about Doom3? Any ideas on Deus Ex2? SS2? Stalker? We have 2/3 games out now that use DX9 true. But are you going to say that those are the end all be all of DX9 performance? We know that Halo allows for the FX to drop down to partial precession. Are all other DX9 games going to do that? That's the issue here. Yes games are much better to gauge hardware. But we all know that every game is different and performance on one game != performance on the other games. 3Dmark is as good as any game that we have now that tries to bench DX performance. Again its one of many things/tools you should use and not the be all end all of benchmarks.
Originally posted by Uttar
AAARGH!:rolleyes:
The compiler is for real guys. Stop inveting BS - everyone is, and it's just annoying me.
The effects of the compiler cannot be as good as hand-tuned code most of the time, but it's pretty darn good. And developers will have making better than what the compiler does.
Disabling the compiler seems like BS though. It's still on, or their scores would be even significantly lower.
Uttar
Uttar,
Even if the "complier" is not as good as the hand crafted stuff, NVIDIA should just take it like mature grown ups and use the compiler. Then we can all stop having to fight about hand crafted shaders being cheating which is exactly what they are (application detection by shader).
If they had used the compiler (assuming it is real) then when FM changed their code, the results would have changed. However, when they changed them again, the next results would be slightly different (less or more optimal). That maybe acceptable to some. IE those who except shader replacement as being valid. However, to change the code all the results flatline appears to be cheating.
If the compiler had kicked in now that the handcrafter shaders for 3dmark2003 cannot be used then it shows us clearly that the compiler is pretty much useless for real world applications. Hence it is a POS.
Why does changing code disable a compiler??? Because it is not a compiler.
Originally posted by aapo
Uhhuh, if you've been reading B3D, you should know the relevant changes in the pixel shaders of the new patch are register permutations. If the nVidia unified compiler would be a real deal, it should have no effect on the performance. Actually, the compiler is a real deal, because performance only drops ~15%. It would be a lot more if the compiler would be the old one - the 15 % difference emerges from hand-compiled shaders replacement (cheating, slightly different output).
Furthermore, someone at B3D extracted the old 3dMark patch 330 shaders, and tried them with his own program. Then he tried the patch 340 shaders with his own program. No performance difference - so nVidia drivers are not only doing shader detection but also application detection to make sure only 3dMark03 shaders get replaced. This is obviously not very nice. :mad:
EDIT: Some clarifications...
Huh?
Not sure where your coming from with this (In regards to my statements).
I wasn't claiming that Futuremark wasn't legitematly trying to block cheats. I believe they are, and I believe nVidia is cheating.
I'm just saying... as long as they take money from IHVs and OEMs... there's always going to be "political accusations", like the one I mentioned. (The can of worms I was referring to).
Though *I* believe Futuremark is doing it's best to stop cheats... others *may* believe something stupid like "Dell threw it's weight around because they're selling ATi cards now". Which, in the consumer's eye, may make 3DMark not be taken as seriously.
That's why I said it's a bad tool, nothing to do with it's technology. ;)
Regards,
Taz
NickSpolec
11-13-03, 03:00 PM
I especially love this quote from Derek Perez, in regards to the performance drop in 3DMark03 on the GFFX's (with the newest patch)
What we expect will happen is that we'll be forced to expend more engineering effort to update our compiler's fingerprinter to be more intelligent, specifically to make it intelligent in its ability to optimize code even when application developers are trying to specifically defeat compilation and optimal code generation.
This just proves that Nvidia cares far more about benchmarks then actual games. How completely, utterly stupid do you have to be to make a comment like this? It was unintentional on Perez's part --- He was trying to make Nvidia the victum once again, ala "Oh, look what FutureMark is going to force us to do... Now we have to devote more resources to cheating because they keep disabling our cheats!"
Hey, Nvidia, hear's an idea... STOP WORRYING ABOUT 3DMARK AND WORRY MORE ABOUT GAMES.
It is just so infuriating.
Nvidia's like an addict, and the 3DMark score is their crack.
digitalwanderer
11-13-03, 03:09 PM
What we expect will happen is that we'll be forced to expend more engineering effort to update our compiler's fingerprinter to be more intelligent, specifically to make it intelligent in its ability to optimize code even when application developers are trying to specifically defeat compilation and optimal code generation.
:lol2: :lol2: :lol2:
Please! :lol:
I can't believe Kyle didn't call nVidia's BS out for the BS it obviously is. :rolleyes:
Originally posted by euan
Why does changing code disable a compiler??? Because it is not a compiler.
As I said in the other thread, don't put words into my mouth please, and even less so if you didn't understand what I meant :) ;)
That part of NVIDIA's UCT ( Unified Compiler Technology ) which are shader replacements should never be called "Compiler", or even "GPU Compiler" as someone from Gainward said. But PR and Marketing did their job, and that's what we got now.
FM disables the "shader replacement" part of UCT. The legit parts remain used. NVIDIA is trying to keep face by claiming it disables UCT as a whole, which it doesn't.
Uttar
Originally posted by Uttar
As I said in the other thread, don't put words into my mouth please, and even less so if you didn't understand what I meant :) ;)
I don't know where I was putting words in your mouth. Usually when I say stuff it goes in their ears. ;)
StealthHawk
11-13-03, 04:01 PM
Kyle's ignorance shines through again.
He obviously doesn't understand what a compiler is or how it should work.
Nor has he taken the time to investigate why NVIDIA's scores dropped and ATI's didn't, and what changes the 340 patch brought over the 330 patch(I'll clue you in Kyle. NVIDIA PR was truthful, the shaders were being detected and replaced by the "compiler." Ask yourself why scores dropped, because there is no way Futuremark can turn off the Unified Shading Compiler in its true form!)
NVIDIA lost 15% in GT2, 10% in GT1 and 30% in GT4. Interesting how they glossed over GT4's drops, as I would think those are the most damning.
Well, at least I have to give Kyle some credit. He hasn't started using any other synthetic benchmarks. So at least his stance of all synthetics are worthless hasn't contradicted itself.
vampireuk
11-13-03, 04:09 PM
I give it about a week:D
vBulletin® v3.7.1, Copyright ©2000-2012, Jelsoft Enterprises Ltd.