Go Back   nV News Forums > Hardware Forums > Benchmarking And Overclocking

Newegg Daily Deals

Reply
 
Thread Tools
Old 05-25-03, 01:34 AM   #1
Browndog
Registered User
 
Join Date: May 2003
Location: Aussieland
Posts: 7
Talking Is this the end of Synthetic Benchmarks?

Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?
__________________
Shoot straight ya bastards!!
(Breaker Morant)
Browndog is offline   Reply With Quote
Old 05-25-03, 01:57 AM   #2
StealthHawk
Guest
 
Posts: n/a
Default

You missed the point.

This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.

What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.

What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.

2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug in Splinter Cell with nvidia cards.

You missed the point.

Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.

Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on
  Reply With Quote
Old 05-25-03, 02:04 AM   #3
silence
 
silence's Avatar
 
Join Date: Jan 2003
Location: Zagreb, Croatia
Posts: 425
Default

i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....
silence is offline   Reply With Quote
Old 05-25-03, 02:25 AM   #4
jjjayb
Registered User
 
Join Date: Jan 2003
Posts: 101
Default

Actually, I've got more confidence in 3dmark03 after all of this. They actually investigate and find optimizations video card companies are using. They disable the optimizations. What game developer is going to look into their time demos and make sure nobody is optimizing them? I'm not talking about games in general, optimizing them is great if you keep the same image quality. I'm talking about optimizations that only benefit the time demos.

Sure it's sad to see the lenghts that the vid card companies will go to in order to inflate benchmark scores, but at least with 3dmark they are on the look out for these types of things. How do we know the video card companies aren't doing the same kind of optimizations on game benchmarks? We don't. What makes you think they won't do the same thing with aquamark? Do you think aquamark would bother taking the time to investigate and make sure they're not? I highly doubt it. Same for any other game engine based benchmark. They can all be cheated on just as easily.
jjjayb is offline   Reply With Quote
Old 05-25-03, 02:49 AM   #5
jAkUp
eat. sleep. overclock.
 
jAkUp's Avatar
 
Join Date: Dec 2002
Location: Chino, California
Posts: 17,744
Default

im glad futuremark took the effort to stop all this cheating...

hey... maybe they should make a punkbuster for 3dmark 2003....
__________________
965xe || evga x58 classified || 3x evga gtx 480 || 6gb g.skill || win7 x64
jAkUp is offline   Reply With Quote
Old 05-25-03, 03:26 AM   #6
Browndog
Registered User
 
Join Date: May 2003
Location: Aussieland
Posts: 7
Thumbs down

I agree that both ATI and Nvidia have cheated or are currently cheating in 3Dmark, but how can we be sure that Futuremark which is closely aligned with a beta member (ATI).
Will act fairly as judge jury and executioner.
__________________
Shoot straight ya bastards!!
(Breaker Morant)
Browndog is offline   Reply With Quote
Old 05-25-03, 04:12 AM   #7
zakelwe
Registered User
 
Join Date: Dec 2002
Posts: 768
Default

If all video company's signed an agreement not to write optimisations at all for any benchmark , backed up by heavy financial penalties, then we could have confidence again.

That, coupled with this bad PR disaster, might just be enough to stop them.

It's the consumers fault for wanting bar charts and graphs. I get carried away when X does 12.4 and Y does 135 and so smashes it into the ground. Bit irrelevant .

Maybe the way to go is :-

" I wondered around and got between 70-85 fps using 1600x1200 2xAA/4xAF so no problem with IQ or speed at this resolution " ....

The problem is you have to know what your video card does at the same level to be able to jusge whether the upgrade is worthwhile and you might not even have that game.

No easy answers ?

Regards

Andy
zakelwe is offline   Reply With Quote
Old 05-25-03, 06:22 AM   #8
Browndog
Registered User
 
Join Date: May 2003
Location: Aussieland
Posts: 7
Thumbs down

Yes there is no easy way to control this type of thing happening because theres money involved.
__________________
Shoot straight ya bastards!!
(Breaker Morant)
Browndog is offline   Reply With Quote

Old 05-25-03, 06:45 AM   #9
Grrrpoop
Wey aye man!
 
Grrrpoop's Avatar
 
Join Date: Jan 2003
Location: Newcastle, UK
Posts: 162
Default

Quote:
Originally posted by Browndog
Yes there is no easy way to control this type of thing happening because theres money involved.
If there are no Synthetic Benchmarks then there's no way to test how current computer hardware will perform in Future games.

That's the whole point of 3Dmark03, and in its day, 3Dmark01..

If you have no way of proving otherwise, it'll just boil down to PR:

IHV 'A' - "Our card will be the fastest in future games!"

IHV 'B' - "NO! Our card will be the fastest in future games!"

Either that or rely on Game Dev's releasing early code which only one IHV has prepared optimised drivers for
__________________
Don't be Care Less with your language
Grrrpoop is offline   Reply With Quote
Old 05-25-03, 06:46 AM   #10
Sazar
Sayonara !!!
 
Sazar's Avatar
 
Join Date: Jan 2003
Location: Austin, Texas
Posts: 9,297
Default

Quote:
Originally posted by silence
i wonder.....did NV really expect that this thing won't be noticed?i mean, how long it took for ppl to see what's going on with their drivers and post that on internet?

so, what's next?and did they do this on purpose?<- knowing how soon ppl will notice....
AFAIK nvidia has yet to officially announce that it has cheated...

ati has come forth saying its renaming of variables or whatever allowed it to optimize.. and those optimizations were still within the overall 3% variance in the scores that futuremark allows (1.9%) v/s 24% for nvidia...

so really nvidia needs to come forth and ADMIT they cheated... some day
Sazar is offline   Reply With Quote
Old 05-25-03, 09:16 AM   #11
silence
 
silence's Avatar
 
Join Date: Jan 2003
Location: Zagreb, Croatia
Posts: 425
Default

well....my point was more like...did NV have in mind something like making 3Dmark useless?cause if they did, then they'll stick with their story and they acchieved their goal. it's bad PR and even worse business politics, but now u can't trust 3Dmark scores and that's something NV wants all the way.

knowing all stunts NV PR did in last ...let's say 6 month....this isn't to unrealistic. i mean....cheat was discovered so soon, not like it was something it can't be detected. and no_access_to_dev version of 3Dmark........sure, maybe not now, but they WERE in beta and they knew it exists......not like there isn't single soul in NV that wasn't aware they will be caught.....

/me being paranoid

Last edited by silence; 05-25-03 at 09:19 AM.
silence is offline   Reply With Quote
Old 05-25-03, 10:57 AM   #12
evilangel
Registered User
 
Join Date: May 2003
Posts: 15
Default Re: Is this the end of Synthetic Benchmarks?

Quote:
Originally posted by Browndog
Considering both ATI and Nvidia have fiddled around with futuremarks benchmark is it going to become obsolete, as most people want to know how GPU's perform in actual games?

Any Comments?
I'm hoping 3DMark goes right down the toilet personally. I never liked it, never will. I just use FRAPS, works fine for me.
evilangel is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


Similar Threads
Thread Thread Starter Forum Replies Last Post
Base mosaic and rotation? happyfunbot NVIDIA Linux 5 07-13-12 12:23 PM
Upcoming Linux Hardware Benchmarks For June News Archived News Items 0 06-09-12 11:10 AM
Diablo III Gaming Benchmarks on Ivy Bridge & Trinity Laptops News Archived News Items 0 05-16-12 10:40 AM
Better Benchmarks for Big Data ' This Week on inside* Publications News Archived News Items 0 05-05-12 02:00 AM
what benchmarks for nv30 / r300/ 350 borntosoul Rumor Mill 10 10-11-02 04:34 PM

All times are GMT -5. The time now is 06:31 AM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.