nV News Forums

 
 

nV News Forums (http://www.nvnews.net/vbulletin/index.php)
-   Benchmarking And Overclocking (http://www.nvnews.net/vbulletin/forumdisplay.php?f=24)
-   -   Is this the end of Synthetic Benchmarks? (http://www.nvnews.net/vbulletin/showthread.php?t=12354)

Browndog 05-25-03 01:34 AM

Is this the end of Synthetic Benchmarks?
 
Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?

StealthHawk 05-25-03 01:57 AM

You missed the point.

This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.

What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.

What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.

2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug in Splinter Cell with nvidia cards.

You missed the point.

Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.

Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on :(

silence 05-25-03 02:04 AM

i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....

jjjayb 05-25-03 02:25 AM

Actually, I've got more confidence in 3dmark03 after all of this. They actually investigate and find optimizations video card companies are using. They disable the optimizations. What game developer is going to look into their time demos and make sure nobody is optimizing them? I'm not talking about games in general, optimizing them is great if you keep the same image quality. I'm talking about optimizations that only benefit the time demos.

Sure it's sad to see the lenghts that the vid card companies will go to in order to inflate benchmark scores, but at least with 3dmark they are on the look out for these types of things. How do we know the video card companies aren't doing the same kind of optimizations on game benchmarks? We don't. What makes you think they won't do the same thing with aquamark? Do you think aquamark would bother taking the time to investigate and make sure they're not? I highly doubt it. Same for any other game engine based benchmark. They can all be cheated on just as easily.

jAkUp 05-25-03 02:49 AM

im glad futuremark took the effort to stop all this cheating...

hey... maybe they should make a punkbuster for 3dmark 2003....:D

Browndog 05-25-03 03:26 AM

I agree that both ATI and Nvidia have cheated or are currently cheating in 3Dmark, but how can we be sure that Futuremark which is closely aligned with a beta member (ATI).
Will act fairly as judge jury and executioner.:(

zakelwe 05-25-03 04:12 AM

If all video company's signed an agreement not to write optimisations at all for any benchmark , backed up by heavy financial penalties, then we could have confidence again.

That, coupled with this bad PR disaster, might just be enough to stop them.

It's the consumers fault for wanting bar charts and graphs. I get carried away when X does 12.4 and Y does 135 and so smashes it into the ground. Bit irrelevant .

Maybe the way to go is :-

" I wondered around and got between 70-85 fps using 1600x1200 2xAA/4xAF so no problem with IQ or speed at this resolution " ....

The problem is you have to know what your video card does at the same level to be able to jusge whether the upgrade is worthwhile and you might not even have that game.

No easy answers ?

Regards

Andy

Browndog 05-25-03 06:22 AM

Yes there is no easy way to control this type of thing happening because theres money involved.:angel2:

Grrrpoop 05-25-03 06:45 AM

Quote:

Originally posted by Browndog
Yes there is no easy way to control this type of thing happening because theres money involved.:angel2:
If there are no Synthetic Benchmarks then there's no way to test how current computer hardware will perform in Future games.

That's the whole point of 3Dmark03, and in its day, 3Dmark01..

If you have no way of proving otherwise, it'll just boil down to PR:

IHV 'A' - "Our card will be the fastest in future games!"

IHV 'B' - "NO! Our card will be the fastest in future games!"

Either that or rely on Game Dev's releasing early code which only one IHV has prepared optimised drivers for :rolleyes:

Sazar 05-25-03 06:46 AM

Quote:

Originally posted by silence
i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....

AFAIK nvidia has yet to officially announce that it has cheated...

ati has come forth saying its renaming of variables or whatever allowed it to optimize.. and those optimizations were still within the overall 3% variance in the scores that futuremark allows (1.9%) v/s 24% for nvidia...

so really nvidia needs to come forth and ADMIT they cheated... some day

silence 05-25-03 09:16 AM

well....my point was more like...did NV have in mind something like making 3Dmark useless?cause if they did, then they'll stick with their story and they acchieved their goal. it's bad PR and even worse business politics, but now u can't trust 3Dmark scores and that's something NV wants all the way.

knowing all stunts NV PR did in last ...let's say 6 month....this isn't to unrealistic. i mean....cheat was discovered so soon, not like it was something it can't be detected. and no_access_to_dev version of 3Dmark........sure, maybe not now, but they WERE in beta and they knew it exists......not like there isn't single soul in NV that wasn't aware they will be caught.....

/me being paranoid

evilangel 05-25-03 10:57 AM

Re: Is this the end of Synthetic Benchmarks?
 
Quote:

Originally posted by Browndog
Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?

I'm hoping 3DMark goes right down the toilet personally. I never liked it, never will. I just use FRAPS, works fine for me.


All times are GMT -5. The time now is 01:35 PM.

Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.