View Full Version : Tim Sweeny on pixel shaders / ATI
I wish ppl would stop closing threads before I get a chance to comment..
Tim Sweeney Da *MAN* at Epic had this to say to Reverend at http://www.beyond3d.com/
quote:
--------------------------------------------------------------------------------
Pixel shaders are functions, taking textures, constants, and texture coordinates as inputs, and producing colors as outputs. Computer scientists have this notion of extensional equality that says, if you have two functions, and they return the same results for all combinations of parameters, then they represent the same function -- even if their implementaions differ, for example in instruction usage or performance.
Therefore, any code optimization performed on a function that does not change the resulting value of the function for any argument, is uncontroversially considered a valid optimization. Therefore, techniques such as instruction selection, instruction scheduling, dead code elimination, and load/store reordering are all acceptable. These techniques change the performance profile of the function, without affecting its extensional meaning.
Optimization techniques which change your function into a function that extensionally differs from what you specified are generally not considered valid optimizations. These sorts of optimizations have occasionally been exposed, for example, in C++ compilers as features that programmers can optionally enable when they want the extra performance and are willing to accept that the meaning of their function is being changed but hopefully to a reasonable numeric approximation. One example of this is Visual C++'s "improve float consistency" option. Such non-extensional optimizations, in all sane programming systems, default to off.
3D hardware is still at a point in its infancy that there are still lots of nondeterministic issues in the API's and the hardware itself, such as undefined amounts of precision, undefined exact order of filtering, etc. This gives IHV's some cover for performing additional optimizations that change the semantics of pixel shaders, though only because the semantics aren't well-defined in the base case anyway. In time, this will all go away, leaving us with a well-defined computing layer. We have to look back and realize that, if CPU's operated as unpredictably as 3D hardware, it would be impossible to write serious software.
While my email to Tim came with an understanding that he may not wish for us to post his answers, Tim said "Please Post!".
--------------------------------------------------------------------------------
What ATi did
quote:
--------------------------------------------------------------------------------
The 1.9% performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.
--------------------------------------------------------------------------------
In other words what ATi did with a slight reorder of shader instructions is *NOT* cheating. What Nvidia did in changing entire shader routines, recompiling everything with Cg, inserting Clip planes and who knows what else they pulled.. *IS* cheating.
Thats why there is a big difference between 1.9% and 24%. 1.9% is well within the tolerance for shader optimization.
What I dont get is, if ATI's instruction re-ordering is a legitimate optimization, and I can well believe it is, and have absolutely no problems with.
Why the hell are they wasting time REMOVING it just to please Futuremark?!
IMO Futuremark should not have released such an easily compromisable benchmarking program. Knowing their own popularity, they had a duty to make sure 3dmark03 was fair and accurate, and as anti-cheatable as possible.
IMO they did not meet these criteria. And what you end up with is companies cheating to get one up on each other.
I'd also like to see some Correctness tests. i.e. a full precision SOFTWARE render engine should be created, that is used to render a selection of scenes. Then the hardware is used to render the identical scene, and graphs can be made to show error tolerenaces between pixel correctness. This would stop companies replacing entire shaders with slightly lower quality ones. They would stick out like a sore thumb in the correctness graphs.
IMO Futuremarks buisness model of actually being in financial agreements with various hardware manufacturers brings other questions to light.
Why would ATI or nvidia want to pay Futuremark to be part of its beta development program? To get better scores of course. They have nothing to win but bechmark wars. Asking for money to be privey to the development details of the benchmark just stinks IMO.
They should make the whole benchmark open source IMO, so everyone can see what its doing internally. i.e. how many extensions are used on one card, and how many on another etc...
Its all gone a bit crap in recent months..
LMAO in todays world NOTHING is cheat proof.
Well thats a pessimistic attitude to take.
Note how the fillrate scores between the anti-cheat 3dmark, and the previous one doesn't change at all. Basically its easy to cheat in a on-rails 3d scene, but its very hard to cheat in rendering a flat quad that doesn't go off the edges of the screen.
Anti-cheating benchmarks are not impossible. You just need to put a little thought into it.
SmuvMoney
05-24-03, 07:57 PM
Nutty, if FutureMark can show in their 3DM2K3 "rulebook" that they don't want their shaders messed with at all, then ATi is making the right decision by following FM's rules. I think FM is trying to trying to prevent a black box optimization of their shaders, which is what ATi pretty much did. ATi improved the speed without changing functionality or IQ, which would normally be a very good thing. However, this may compromise the workload aspect that FM wanted to impart on this benchmark. I can't blame FM to being very strict after rampant cheating and hacks in 2K1. They are protecting integirty of the benchmark and their IP. In any other circumstance, ATi's optimization is completely legit (Tim's point).
StealthHawk
05-24-03, 08:29 PM
Originally posted by Nutty
What I dont get is, if ATI's instruction re-ordering is a legitimate optimization, and I can well believe it is, and have absolutely no problems with.
Why the hell are they wasting time REMOVING it just to please Futuremark?!
I don't think they are removing it to please Futuremark. Futuremark said they are investigating the issue further, they did not come to the conclusion that ATI was cheating yet.
I think ATI's move is twofold.
1) There are people who consider this cheating, regardless of whether it is functionally the same, or whether Tim Sweeney says it is legitmate optimization.
Discussion (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=12311)
Discussion (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=12299)
Read through either of those threads, and you will see that people consider ATI's moves cheating.
2) This makes them look responsible. While their competitor does not admit to cheating, does not say they will remove cheating code from drivers, and resorts to attacking Futuremark, ATI owns up to thier "legitmate but questionable" optimizations and says they will remove them so there will be no controversy.
Anti-cheating benchmarks are not impossible. You just need to put a little thought into it.
A LITTLE thought into it? Are you insane? I can see myself making a benchmark proggie and taking 90% of the time trying to make it cheat proof. Then constantly patching the proggie from future cheats in drivers and/or 3rd party proggies.
As long as there is a will to cheat, nothing will ever be 100% cheat proof.
There are people who consider this cheating, regardless of whether it is functionally the same, or whether Tim Sweeney says it is legitmate optimization.
Just cos a load of ppl whinge when the fact is they haven't a clue what they're talking about, is enough cause for a 3d company to strip out its LEGITMATE optimizations?
Its just the same as C compilers re-arranging your order of execution to improve efficiency on the hardware.
You dont see ppl complaining about that do you?
Fair enuff if they're gonna remove it, but that means one of two things;
a) We now have a 3dmark check in ATI's drivers turning off optimization. Apps specific checks are bad Mmmmmkay?
b) Or we've lost that optimization entirely, so ATI owners can expect a nice performance drop in their next drivers.
Nice.
What a bunch of muppets, all over something you can't even play in the 1st place.
:rolleyes:
Originally posted by Nutty
Just cos a load of ppl whinge when the fact is they haven't a clue what they're talking about, is enough cause for a 3d company to strip out its LEGITMATE optimizations?
Its just the same as C compilers re-arranging your order of execution to improve efficiency on the hardware.
You dont see ppl complaining about that do you?
Fair enuff if they're gonna remove it, but that means one of two things;
a) We now have a 3dmark check in ATI's drivers turning off optimization. Apps specific checks are bad Mmmmmkay?
b) Or we've lost that optimization entirely, so ATI owners can expect a nice performance drop in their next drivers.
Nice.
What a bunch of muppets, all over something you can't even play in the 1st place.
:rolleyes:
the drop in score == > 2% overall so not really something to worry about :)
basically it is a good move... they explained what was done and that they are removing it and therefore consumers from both parties (and matrox and others) will not point fingers @ ati...
notice the amount of noise this has created in relation to the various 'new' findings about nvidia on a daily basis it would seem..
nvidia's refusal to address this matter in a timely and mature manner has lead to more and more accusations flying... concerning other benchmarks and this can only bugger a company's reputation...
though if it was nvidia's reputation in terms of ethics @ this point... it can't really get much lower...
Hanners
05-25-03, 08:05 AM
Originally posted by Nutty
Fair enuff if they're gonna remove it, but that means one of two things;
a) We now have a 3dmark check in ATI's drivers turning off optimization. Apps specific checks are bad Mmmmmkay?
b) Or we've lost that optimization entirely, so ATI owners can expect a nice performance drop in their next drivers.
Nice.
You don't seem to have realised the fact that ATi's drievrs are only reordering the shader instructions specifically for GT4 in 3DMark 2003. They won't need a 3DMark check to turn off the optimisation when they can just remove it.
The only thing ATi owners will experience a performance drop in is 3DMark 2003, and even then only to the tune of 1-2% overall. Not quite a 25% drop by any means.
You don't seem to have realised the fact that ATi's drievrs are only reordering the shader instructions specifically for GT4 in 3DMark 2003. They won't need a 3DMark check to turn off the optimisation when they can just remove it.
from ATI's official response they worded it to sound like the re-ordering was automatic. It was not manually done for the sole case of 3dmark.
If this is the case, then removing it is bad, as its not a 3dmark specific optimization.
no it's not a generic optimisation, it's for 3dmark only.
Then they're no better than nvidia then.
StealthHawk
05-25-03, 07:50 PM
Originally posted by Nutty
from ATI's official response they worded it to sound like the re-ordering was automatic. It was not manually done for the sole case of 3dmark.
If this is the case, then removing it is bad, as its not a 3dmark specific optimization.
ATI was detecting 3dmark to turn on the optimization. It wasn't a global driver optimization. Otherwise the new patch wouldn't have fudged the 8% optimization in GT4.
So...now that you know it was 3dmark specific you are changing your stance to what ATi cheated?
Originally posted by Nutty
from ATI's official response they worded it to sound like the re-ordering was automatic. It was not manually done for the sole case of 3dmark.
If this is the case, then removing it is bad, as its not a 3dmark specific optimization.
ati made it clear it was 3dmark specific... ie reference to sky and water shaders... :)
therefore it WAS application specific.. but seeing as how this optimization could be viewed as cheating... ati is voluntarily removing it...
of course since it is detecting the application and therefore changing to compensate for the app... it can easily be considered cheating... but the way it works is VERY different from the way nvidia's cheats work... though it should not have been there in the first place and per the criterion placed on nvidia's drivers and the meaning of the word CHEAT... ati did in fact commit some cheating here... not IMO but I am sure many people will see it as such...
btw.. changing things to take better advantage of ones hardware while @ the same time maintaining precision and IQ levels @ reference levels and not changing the overall level of work done is something that could be considered a valid optimization :) per a coupla peeps in the know :)
the overall score variance was 1.9%... which is well within the range specified by futuremark of 3%...
compare this with 20+ % variance for nvidia...
compare this further with nvidia's response...
Originally posted by Nutty
Then they're no better than nvidia then.
Well, considering they are a large company trying to make profit and take as much market share as possible and to do that they must make themselves look better than the competition in any way they can. What else do you expect?
A corporation with morals and ethics? Now that's funny. :lol:
heh yeah I take it back, ATI sucks! :)
The way I 1st read it made it look like it was a global optimization, my bad.
oh well..
no thinking in anyone here in particular..
but it really sucks ,when journalists put WOrds in the mouth of people
they have never told .
TIm sweeney of EPIC have never Told ATI was cheating or not cheating.
neither that Nvidia was cheating or not cheating.
he simply told what in his opinion is a cheat or not a cheat in games.
but EPIC will never claim to know who is cheating or who is not.
Only FUturemark can says that . they are the only ones that set the rules in their benchmark . ATI "optimizations" or cheatings can be valid in a game or not ,but if Futuremark says thats is ilegal in their benchmark ,then it is a cheat.
ATI optimizations" as ET (http://www.extremetech.com/article2/0,3973,1105259,00.asp) says dropped the scores up to 11% in game4.
so real optimizations will never drop the performance by so much.
(i dont care if the scores drop 500% -3dmark is useless for me so ati or Nvidia can cheat or optimize all they want)
but the issue is that i see people quoting the "opinions" of hardware sites (ATI biased sites) , as the ultimate WOrd of who is "guilty" and " who is "inocent". :rolleyes:
Hanners
05-27-03, 03:33 PM
Originally posted by Nv40
it really sucks ,when journalists put WOrds in the mouth of a game developer ,that he has never told..
TIm sweeney of EPIC have never Told ATI was cheating or not cheating.
neither that Nvidia was cheating or not cheating.
he simply told what in his opinion is a cheat or not a cheat in games.
I'm sure Tim Sweeney didn't wake up the other morning and think out of the blue 'You know what? I think I'll talk to Beyond3D about driver optimisations today!'. He was commenting on the 3DMark situation, like it or not.
While I would say that the rules of optimisation differ between benchmarks and games (to my mind at least), what Tim had to say was a very valid and useful point in this whole discussion, and isn't one that should be dismissed.
R.Carter
05-27-03, 03:36 PM
Originally posted by Nutty
IMO Futuremark should not have released such an easily compromisable benchmarking program. Knowing their own popularity, they had a duty to make sure 3dmark03 was fair and accurate, and as anti-cheatable as possible.
Well... the problem is the cheating is going on at the hardware level and there is no real way that the application can detect that the driver has pulled a fast one.
I mean it sends a shader program to the driver and the driver sees that it matches one that it has an optimized version of and it uses its shader instead of the one the program gave it.
Just how is the program going to know that this has happened?
The only way is to take screenshots and then compare the two screenshots pixel for pixel and see if there are any differences. In the case of ATI there wouldn't have been any visual differences so even they you couldn't detect that a swap has taken place.
Originally posted by Hanners
While I would say that the rules of optimisation differ between benchmarks and games (to my mind at least), what Tim had to say was a very valid and useful point in this whole discussion, and isn't one that should be dismissed.
what Epic have told ,its 100% true ,about his games .
those are Epic rules or opinions about what is valid or not.
but another gamedeveloper can say..
"if the IQ is pretty similar in my games ,and the performance is good,
then i dont care what cheat/hack or optimization is done ,what really
matters to me is the gamer experience"
just notice that many games dont have a TIMEDEMO.
means those developers doesnt care about video card comparisons.
for those developers is irrelevant is ATI or Nvidia rewrite the entire game if the game IQ pretty similar for the ->gamers ,and the performance is great ,and the gamedeveloper is happy ,then
any hack/optimization or cheat is valid for them.
but the issue here is not Gamedevelopers comments , Epic comments
are very usefull ,the issue is when people Link Epic comments
with the comments of ATI . in a way (suggesting) that Epic was validating
ATI behaviours in 3dmark . when probably they have no clue ,or have not
being informed well about the details,of ATi comments or about what ATi have done in 3dmark .
it just remember the war of Iraq.. :)
the pentagon shows the videos of the accuracy of their weapons
and Arabs channels show women and childrens dead.
playing with the evidence ,to manipulate the public opinion ,for their personal agendas is an example of How not to tell the truth to the public.
Dont drag the Freakin' Iraqi war into a damn video card discussion. Of course women and children were killed.
ITS A WAR. Do you really think Americans are so stupid as to believe that no innnocents died?.....drop this stupid analogy you just brought up.
Originally posted by micron
Dont drag the Freakin' Iraqi war into a damn video card discussion. Of course women and children were killed.
ITS A WAR. Do you really think Americans are so stupid as to believe that no innnocents died?.....drop this stupid analogy you just brought up.
hehe...
there is no point to discuss about the IRaq war here... :)
already have finished. forget the example :)
the point is that i see JOurnalims here
used to manipulate the public opinion about something ,without
giving the public a way to form their own conclusions .
but only the ones that benefits only their personal interests.
EPic and Idsoftware are 100% correct. the thing is 3dmark is not a game.
are their are complaining because X and Y company is not doing things
in the same way was requested. (Even if the output is the same).
do i care? not i dont , the more both cheats ,the more people will
stop using the benchmark. :D
but its simply stupid to justify the "optimizations"
of X company ,and critisize the "optimizations" of Y company.
email gamedevelopers and "interpret" their words as X company guilty
and Y company inocent. everything is about the RULES of 3dmark.
if the benchmark is stupid ,and ask stupid things ,then both companies
need to follow its stupidity . and stupid gamers will buy video cards just for that. when clearly Both companies names are in the same REport of
driver irregularities or aka (cheating) .if X company is removing their "optimizations" is because FM doesnt want them.
........(Picks up big rock and throws it at NV40)
StealthHawk
05-27-03, 11:38 PM
Originally posted by micron
........(Picks up big rock and throws it at NV40)
The ignore feature of vbulletin messageboards is very helpful ;)
Originally posted by Nv40
hehe...
there is no point to discuss about the IRaq war here... :)
already have finished. forget the example :)
the point is that i see JOurnalims here
used to manipulate the public opinion about something ,without
giving the public a way to form their own conclusions .
but only the ones that benefits only their personal interests.
EPic and Idsoftware are 100% correct. the thing is 3dmark is not a game.
are their are complaining because X and Y company is not doing things
in the same way was requested. (Even if the output is the same).
do i care? not i dont , the more both cheats ,the more people will
stop using the benchmark. :D
but its simply stupid to justify the "optimizations"
of X company ,and critisize the "optimizations" of Y company.
email gamedevelopers and "interpret" their words as X company guilty
and Y company inocent. everything is about the RULES of 3dmark.
if the benchmark is stupid ,and ask stupid things ,then both companies
need to follow its stupidity . and stupid gamers will buy video cards just for that. when clearly Both companies names are in the same REport of
driver irregularities or aka (cheating) .if X company is removing their "optimizations" is because FM doesnt want them.
it is quite amusing to see you call other people stupid :D
ok.. ok I take it back :D... i just couldn't resist...
but seriously.. why are gamers 'stupid' for buying cards based on the scores of 3dmark03 ? I am sure you had no problems with people buying nvidia cards based on 3dmark2001... :)
vBulletin® v3.7.1, Copyright ©2000-2012, Jelsoft Enterprises Ltd.