Is this the end of Synthetic Benchmarks?

[Browndog]

Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?

[StealthHawk]

You missed the point.

This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.

What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.

What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.

2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=12339) in Splinter Cell with nvidia cards.

You missed the point.

Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.

Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on :(

[silence]

i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....

[jjjayb]

Actually, I've got more confidence in 3dmark03 after all of this. They actually investigate and find optimizations video card companies are using. They disable the optimizations. What game developer is going to look into their time demos and make sure nobody is optimizing them? I'm not talking about games in general, optimizing them is great if you keep the same image quality. I'm talking about optimizations that only benefit the time demos.

Sure it's sad to see the lenghts that the vid card companies will go to in order to inflate benchmark scores, but at least with 3dmark they are on the look out for these types of things. How do we know the video card companies aren't doing the same kind of optimizations on game benchmarks? We don't. What makes you think they won't do the same thing with aquamark? Do you think aquamark would bother taking the time to investigate and make sure they're not? I highly doubt it. Same for any other game engine based benchmark. They can all be cheated on just as easily.

[jAkUp]

im glad futuremark took the effort to stop all this cheating...

hey... maybe they should make a punkbuster for 3dmark 2003....:D

[Browndog]

I agree that both ATI and Nvidia have cheated or are currently cheating in 3Dmark, but how can we be sure that Futuremark which is closely aligned with a beta member (ATI).
Will act fairly as judge jury and executioner.:(

[zakelwe]

If all video company's signed an agreement not to write optimisations at all for any benchmark , backed up by heavy financial penalties, then we could have confidence again.

That, coupled with this bad PR disaster, might just be enough to stop them.

It's the consumers fault for wanting bar charts and graphs. I get carried away when X does 12.4 and Y does 135 and so smashes it into the ground. Bit irrelevant .

Maybe the way to go is :-

" I wondered around and got between 70-85 fps using 1600x1200 2xAA/4xAF so no problem with IQ or speed at this resolution " ....

The problem is you have to know what your video card does at the same level to be able to jusge whether the upgrade is worthwhile and you might not even have that game.

No easy answers ?

Regards

Andy

[Browndog]

Yes there is no easy way to control this type of thing happening because theres money involved.:angel2:

[Grrrpoop]

Originally posted by Browndog
Yes there is no easy way to control this type of thing happening because theres money involved.:angel2:
If there are no Synthetic Benchmarks then there's no way to test how current computer hardware will perform in Future games.

That's the whole point of 3Dmark03, and in its day, 3Dmark01..

If you have no way of proving otherwise, it'll just boil down to PR:

IHV 'A' - "Our card will be the fastest in future games!"

IHV 'B' - "NO! Our card will be the fastest in future games!"

Either that or rely on Game Dev's releasing early code which only one IHV has prepared optimised drivers for :rolleyes:

[Sazar]

Originally posted by silence
i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....

AFAIK nvidia has yet to officially announce that it has cheated...

ati has come forth saying its renaming of variables or whatever allowed it to optimize.. and those optimizations were still within the overall 3% variance in the scores that futuremark allows (1.9%) v/s 24% for nvidia...

so really nvidia needs to come forth and ADMIT they cheated... some day

[silence]

well....my point was more like...did NV have in mind something like making 3Dmark useless?cause if they did, then they'll stick with their story and they acchieved their goal. it's bad PR and even worse business politics, but now u can't trust 3Dmark scores and that's something NV wants all the way.

knowing all stunts NV PR did in last ...let's say 6 month....this isn't to unrealistic. i mean....cheat was discovered so soon, not like it was something it can't be detected. and no_access_to_dev version of 3Dmark........sure, maybe not now, but they WERE in beta and they knew it exists......not like there isn't single soul in NV that wasn't aware they will be caught.....

/me being paranoid

[evilangel]

Originally posted by Browndog
Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?

I'm hoping 3DMark goes right down the toilet personally. I never liked it, never will. I just use FRAPS, works fine for me.

[evilangel]

Originally posted by silence
well....my point was more like...did NV have in mind something like making 3Dmark useless?cause if they did, then they'll stick with their story and they acchieved their goal. it's bad PR and even worse business politics, but now u can't trust 3Dmark scores and that's something NV wants all the way.



That's actually a good point. If Nvidia proves it's easy to cheat but don't admit it, it may work the other way and hurt FutureMark. It may make enough people paranoid about what's what that they may say screw it with synthetic benches. Although FutueMArk patched the bench, what's to say Nvidia won't just do it again in a smarter way.

Also this could be where Nvidia might want to create their own standards, like Microsoft. This is where people get bent out of shape. Some people say stick to the standards that are in place. If Nvidia wants to create a different way of doing things that benefits the games being played on their cards, doesn't bother me. If you don't like it, don't buy an Nvidia card.

Personally, i'm wondering if there's just bad blood between FutureMark and Nvidia at this point just for the fact that Nvidia didn't pay the couple of hundred thousand dollars to participate in the program.

I think new games coming out should just have a playable demo you can download a few months before it arrives that incorporates native benching and that's that. This way you'll know how the game is going to run on your card before it comes out.

The whole 3DMark thing just never appealed to me, it seems like it's more a sport now than what it was intended to be.

[Morrow]

Originally posted by evilangel
Although FutueMArk patched the bench, what's to say Nvidia won't just do it again in a smarter way.


Why should they do it in a smarter way if they deliberately want the cheats to be found? Maybe this is really their goal, to screw FutureMark with continously releasing cheats in their drivers to synthetically increase the score for their cards?

I mean, 3dmark03 is obviously an unfair benchmark. FM wants to bench standard performance however games use optimized paths where nvidia hardware shines (like with SSE2 optimizations on P4, the P4 is unbeatable with SSE optimized instructions, that's the reason why AMD was so interested in integrating SSE2 in their Hammer CPUs). The R3x0 has more raw power which favorites Radeons in 3dmark03.

Adding to that, the fact that Radeons use the faster FP24 mode in contrast to the much slower FP32 but higher IQ mode of the GeForces doesn't say anything positive about the fairness of this bench. When nvidia used FP16 for 3dmark03 they were called cheater because they used lower precision. Now that nvidia is using FP32 and ATI still using the faster FP24 because it's the only mode they support, does that make ATI also cheater because they are using a lower precision than nvidia?

Hopefully you have noticed by now that 3dmark03 can impossible be a fair benchmark. The current two biggest architectures (R3x0 and nv3x) are too different to create a fair benchmark. Synthetic benchmarks are ending here...

[evilangel]

Originally posted by Morrow
Why should they do it in a smarter way if they deliberately want the cheats to be found? Maybe this is really their goal, to screw FutureMark with continously releasing cheats in their drivers to synthetically increase the score for their cards?

I mean, 3dmark03 is obviously an unfair benchmark. FM wants to bench standard performance however games use optimized paths where nvidia hardware shines (like with SSE2 optimizations on P4, the P4 is unbeatable with SSE optimized instructions, that's the reason why AMD was so interested in integrating SSE2 in their Hammer CPUs). The R3x0 has more raw power which favorites Radeons in 3dmark03.

Adding to that, the fact that Radeons use the faster FP24 mode in contrast to the much slower FP32 but higher IQ mode of the GeForces doesn't say anything positive about the fairness of this bench. When nvidia used FP16 for 3dmark03 they were called cheater because they used lower precision. Now that nvidia is using FP32 and ATI still using the faster FP24 because it's the only mode they support, does that make ATI also cheater because they are using a lower precision than nvidia?

Hopefully you have noticed by now that 3dmark03 can impossible be a fair benchmark. The current two biggest architectures (R3x0 and nv3x) are too different to create a fair benchmark. Synthetic benchmarks are ending here...

Good points. We agree that we don't like 3DMark and it's useles.
I think you will see ATI and Nvidia growing more apart and doing things their own way.

[SlyBoots]

Originally posted by evilangel
Good points. We agree that we don't like 3DMark and it's useles.
I think you will see ATI and Nvidia growing more apart and doing things their own way.

In reference to Kyle's comment that 3dmark03 was usless, Kristof wrote>

"Note how he talks about the overall score of 3DMark, I think everybody agrees that the total score as reported by 3DMark has no real value - the detail scores however are very valuable and there is no way to possibly claim that they are not useful. Afterall when do you expect that a game will come out with the shader load available in 3DMark today and decent benchmarking functionality ? They keep talking about Quake3, which for todays graphics hardware is pretty much turning into a CPU test (how low can your driver overhead go ?).

In a real game how is he going to check each shader ? Can he run every game using the ref rast so he can judge if the correct accuracy is being maintained and not some hacked looking similar shader ?

All in all 3DMark is a very valuable test set, the score is just a number which indicates that higher is better and it satisfies the most basic user that only cares to check if his system is performing roughly as it should - it also satisfies the tweak freaks so they can battel for the highest scores. "

[StealthHawk]

Originally posted by Morrow
Why should they do it in a smarter way if they deliberately want the cheats to be found? Maybe this is really their goal, to screw FutureMark with continously releasing cheats in their drivers to synthetically increase the score for their cards?

Along with making Futuremark and 3dmark03 look bad, nvidia has also raised the possibility of cheating in EVERY benchmark(including game benchmarks), as well as cheating in shader programs used in games.

Are we really supposed to be reassured by this?

Adding to that, the fact that Radeons use the faster FP24 mode in contrast to the much slower FP32 but higher IQ mode of the GeForces doesn't say anything positive about the fairness of this bench. When nvidia used FP16 for 3dmark03 they were called cheater because they used lower precision. Now that nvidia is using FP32 and ATI still using the faster FP24 because it's the only mode they support, does that make ATI also cheater because they are using a lower precision than nvidia?

Sigh. Look at a real world example. John Carmack said that NV30 ran the ARB2 path 50% slower than an R300 did. NV30 is using FP32, and R300 is using FP24. Now ask yourself why a 33% increase in precision would drop performance by 50%?

You also disregard some facts such as FP24 is the minimum precision required by DX9. No one told nvidia to support FP32. No one mad nvidia not support FP24 AND FP32. Dropping down to FP16(which nvidia did in 3dmark03) means that nvidia is cheating, because they are no longer in the DX9 spec. Sorry, nvidia has no one to blame but themselves.

[StealthHawk]

Originally posted by Morrow
I mean, 3dmark03 is obviously an unfair benchmark. FM wants to bench standard performance however games use optimized paths where nvidia hardware shines (like with SSE2 optimizations on P4, the P4 is unbeatable with SSE optimized instructions, that's the reason why AMD was so interested in integrating SSE2 in their Hammer CPUs). The R3x0 has more raw power which favorites Radeons in 3dmark03.


And from a developer's perspective: I find it interesting that Futuremark works with ATI and Nvidia to make the benchmark run as fast as possible on specific hardware. Game developers don't have the luxury of doing this. Publishers want them to finish the game as quick as possible and most of them don't have months to spend tweaking code paths for specific cards, especially when new cards are coming out every few months.

I think Nvidia saying that Futuremark is out to get them is nonsense. An application should run faster than your competition regardless of how it was programmed. Game developers aren't going to design games that match your hardware's abilities exactly, in terms of number of textures used, shader instructions, etc. Picking apart the techniques used in 3DMark is rather pointless. You might as well pick at every game and complain how they didn't optimize for your card.

3DMark should be used by Nvidia to tell where their card needs improvement, not as an advertisement to make their cards look good.

http://www.beyond3d.com/forum/viewtopic.php?t=6087

[Sazar]

Originally posted by StealthHawk
Along with making Futuremark and 3dmark03 look bad, nvidia has also raised the possibility of cheating in EVERY benchmark(including game benchmarks), as well as cheating in shader programs used in games.

Are we really supposed to be reassured by this?

Sigh. Look at a real world example. John Carmack said that NV30 ran the ARB2 path 50% slower than an R300 did. NV30 is using FP32, and R300 is using FP24. Now ask yourself why a 33% increase in precision would drop performance by 50%?

You also disregard some facts such as FP24 is the minimum precision required by DX9. No one told nvidia to support FP32. No one mad nvidia not support FP24 AND FP32. Dropping down to FP16(which nvidia did in 3dmark03) means that nvidia is cheating, because they are no longer in the DX9 spec. Sorry, nvidia has no one to blame but themselves.

those are pretty much my thoughts on this matter :(

the pixel shader/vertex shader performance is lower than one should expect... and IMO should be questioned and FIXED by nvidia before they decide to tape out other gpu's... better ps/vs performance == ati will have to work harder as well due to their having to compete == consumers win...

[Morrow]

Originally posted by StealthHawk
Sigh. Look at a real world example. John Carmack said that NV30 ran the ARB2 path 50% slower than an R300 did. NV30 is using FP32, and R300 is using FP24. Now ask yourself why a 33% increase in precision would drop performance by 50%?


Did you read my post?

I said optimizations is the aspect where nvidia hardware shines, not standard instructions (like the ARB2)! We all know that the nv3x is slow in ARB2 but we also all know that Doom3 won't use ARB2 to render the graphics on nv3x hardware. So what is the problem besides 3dmark03 using standard code?

Isn't it obvious, the nv3x is faster in real-life games (except in SplinterCell) and the Radeons are faster in 3dmark03. So what do you want to play? games or 3dmark?

Anyway, if 16-bit color (65535 colors) is twice the precision of 8-bit (256 colors), yes , in that case FP32 is 33% more precision than FP24. Just why does FP32 have 256 times more colors available for rendering than FP24? It's certainly only for marketing like 32-bit colors in current games :)

[Hanners]

Originally posted by Morrow
Did you read my post?

I said optimizations is the aspect where nvidia hardware shines, not standard instructions (like the ARB2)! We all know that the nv3x is slow in ARB2 but we also all know that Doom3 won't use ARB2 to render the graphics on nv3x hardware. So what is the problem besides 3dmark03 using standard code?

I think StealthHawk's point is that most games aren't optimised for particular hardware because developers simply don't have time to, as shown in the quote StealthHawk made from the developer over at Beyond3D.

[StealthHawk]

Originally posted by Morrow
Did you read my post?

I said optimizations is the aspect where nvidia hardware shines, not standard instructions (like the ARB2)! We all know that the nv3x is slow in ARB2 but we also all know that Doom3 won't use ARB2 to render the graphics on nv3x hardware. So what is the problem besides 3dmark03 using standard code?

Isn't it obvious, the nv3x is faster in real-life games (except in SplinterCell) and the Radeons are faster in 3dmark03. So what do you want to play? games or 3dmark?

No, that's not what you said at all. You said "games use optimized paths where nvidia hardware shines." You also bring up AMD vs Intel so you are implying that most games will have specific paths which are optimized for each card.

This is not true of most games. Sure, someone like Carmack has the ability, time, and money to finish a game engine to his liking. Most most devs don't have his luxury.

Just why does FP32 have 256 times more colors available for rendering than FP24?

I'm not sure if that number is correct, so I won't comment on it.

It's certainly only for marketing like 32-bit colors in current games :)

:confused: Now you confuse me. Are you saying that 32bit color in games is unnecessary? There is a vast difference between 16bit color and 32bit color in games today. There wasn't back when 32bit color was introduced, because the art was not tuned for 32bit color.

[evilangel]

I think you'll see more and more games optimized for either ATI or Nvidia in the future. Whoever sells more cards and has more market share will have more games optimized for them, just like with consoles. I think standards are going to go down the toliet.

[Morrow]

Originally posted by StealthHawk
You also disregard some facts such as FP24 is the minimum precision required by DX9. No one told nvidia to support FP32. No one mad nvidia not support FP24 AND FP32. Dropping down to FP16(which nvidia did in 3dmark03) means that nvidia is cheating, because they are no longer in the DX9 spec. Sorry, nvidia has no one to blame but themselves.

FP24 minimum for DX9? Please explain the following quote:

"DX9 and ARB_fragment_program assume 32 bit float operation, and ATI just converts everything to 24 bit."

Isn't that exactly what I was saying, namely that ATI is rendering in a lower precision than they are supposed to and "assumed by DX9". So my question if ATI is cheating since they are using the faster FP24 mode and nvidia the slower but higher precision FP32 mode was not that wrong to ask after all :)

In fact, I see a conspiracy raising here which basically indicates that ATI deliberately only added FP24 mode to their R3x0 core so no one can force them to use another FP mode for DX9 compatibility. Nvidia on the contrary wanted to add flexibility to their nv3x architecture but now is forced by MS to default to FP32 if not requested otherwise by the game/program. Of course, speed-wise FP32 can't keep up with FP24 eventhough it offers a better IQ...

BTW, the quote from above is from no other person than John Carmack himself, posted today at slashdot.org (http://slashdot.org/comments.pl?sid=65617&cid=6051216)

[Hanners]

Originally posted by Morrow
FP24 minimum for DX9? Please explain the following quote:

"DX9 and ARB_fragment_program assume 32 bit float operation, and ATI just converts everything to 24 bit."

Isn't that exactly what I was saying, namely that ATI is rendering in a lower precision than they are supposed to and "assumed by DX9". So my question if ATI is cheating since they are using the faster FP24 mode and nvidia the slower but higher precision FP32 mode was not that wrong to ask after all :)

Actually, if I remember correctly the entire R300 pipeline supports FP32 (128-bit), apart from the Pixel Shaders which are FP24 (96-bit).

Considering that NV30 has to lower the quality for everything to FP16 or FX12 to avoid cutting its performance in half, you could spin the whole 'whose cheating now?' argument around again.

At the end of the day, nobody is cheating in the sense of what precision they are using, all that's happened is that both IHVs have chosen different strategies for their architectures, both of which have their good and bad points.