Go Back   nV News Forums > Hardware Forums > Benchmarking And Overclocking

Newegg Daily Deals

Reply
 
Thread Tools
Old 09-15-03, 09:42 PM   #145
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Quote:
Originally posted by The Baron
Better idea--define what entails a "cheat" and what entails a legitimate "optimization." Then have that conversation.
An optimization is something that adds to performance without a loss of image quality. A cheat is an optimization that does reduce image quality at the cost of performance or violates the rules of a benchmarking environment (clip planes and such ).

Trade-offs between image quality for performance CAN be considered optimizations as long as the user has control over them, like a "boost" or "suck-arse-graphics-for-speed" button or something...but it has to be a choosable option.

(Oh, and any optimizations that over-ride a users preferences are no-no's too. )
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Old 09-15-03, 09:50 PM   #146
Evan Lieb
Registered User
 
Join Date: Sep 2003
Posts: 38
Default

Quote:
Originally posted by digitalwanderer
What would you suggest?
It's tough to say, though off the top of my head the [H], AT, THG, B3D, etc. would have to agree on:

1. What constitutes a cheat and what constitutes an optimization? Is sacrificing a tiny bit of IQ in an First Person Shooter game that only a screenshot would show really a cheat if fps go up by 50% from 30 to 45? What if fps only go up by 10% from 30 to 33 and IQ degradation is noticeable during gameplay in that same FPS game?

2. How to deal with previews, early review samples, etc. in regards to how the product will actually perform once they reach customers with final silicon, final drivers, etc. How to balance exclusives and early review samples in other words.

Among other things. I'll run it by Anand and see what he thinks.
Evan Lieb is offline   Reply With Quote
Old 09-15-03, 09:53 PM   #147
indio
Registered User
 
Join Date: May 2003
Posts: 116
Default

Quote:
Originally posted by The Baron
Better idea--define what entails a "cheat" and what entails a legitimate "optimization." Then have that conversation.
Legit Optimization = Faster and/or smoother performance without degrading image quality relative to previous driver versions and the DX reference rasterizer. If image quality is degraded it should only be done to make an unacceptably performing application usable and should be forthcoming that this was done in the release notes.

Cheat = ANYTHING ELSE!
indio is offline   Reply With Quote
Old 09-15-03, 09:56 PM   #148
Sazar
Sayonara !!!
 
Sazar's Avatar
 
Join Date: Jan 2003
Location: Austin, Texas
Posts: 9,297
Default

Quote:
Originally posted by Evan Lieb
It's tough to say, though off the top of my head the [H], AT, THG, B3D, etc. would have to agree on:

1. What constitutes a cheat and what constitutes an optimization? Is sacrificing a tiny bit of IQ in an First Person Shooter game that only a screenshot would show really a cheat if fps go up by 50% from 30 to 45? What if fps only go up by 10% from 30 to 33 and IQ degradation is noticeable during gameplay in that same FPS game?

2. How to deal with previews, early review samples, etc. in regards to how the product will actually perform once they reach customers with final silicon, final drivers, etc. How to balance exclusives and early review samples in other words.

Among other things. I'll run it by Anand and see what he thinks.
do you really think [H], AT and THG are qualified to be considered 'referees' in a situation such as this just looking over their history ?

it would be nice to be able to give a website the benefit of the doubt but when various reviews come out with information and conclusions that completely contradict what is being seen by many other people and is later proven to be less than sturdy information... that website loses credit...

just look @ [H]... a decent website with decent reviews for various items... but when it comes to gfx cards... their credibility is not much to talk about
@ this time...

as for THG... well they have lars... nuff said...

AT used to be one of the places I used to go to but the unaddressed and very slow to be resolved issue of testing concerning Splinter Cell (with filtering) and the bogus conclusions drawn.. as well as a few other tests... have left a sour taste in the mouth...

sry... had to let out that little rant...
Sazar is offline   Reply With Quote
Old 09-15-03, 10:00 PM   #149
Evan Lieb
Registered User
 
Join Date: Sep 2003
Posts: 38
Default

I see what you guys mean, but there are grey areas that are hard to judge (like the first scenario I listed in my previous post) that happen quite consistently, but might not necessarily be a big deal. Of course, then you have the problem of too general of a maxim; that is, perhaps some minor IQ degradation is OK if fps go up noticeably in a First Person Shooter game, but in a more slow paced game (like an RTS) where speed isn't as important and IQ, therefore, becomes the primary focus, that maxim falls flat on its face.
Evan Lieb is offline   Reply With Quote
Old 09-15-03, 10:01 PM   #150
Evan Lieb
Registered User
 
Join Date: Sep 2003
Posts: 38
Default

Quote:
Originally posted by Sazar
do you really think [H], AT and THG are qualified to be considered 'referees' in a situation such as this just looking over their history ?

it would be nice to be able to give a website the benefit of the doubt but when various reviews come out with information and conclusions that completely contradict what is being seen by many other people and is later proven to be less than sturdy information... that website loses credit...

just look @ [H]... a decent website with decent reviews for various items... but when it comes to gfx cards... their credibility is not much to talk about
@ this time...

as for THG... well they have lars... nuff said...

AT used to be one of the places I used to go to but the unaddressed and very slow to be resolved issue of testing concerning Splinter Cell (with filtering) and the bogus conclusions drawn.. as well as a few other tests... have left a sour taste in the mouth...

sry... had to let out that little rant...
Everyone's entitled to their opinion.
Evan Lieb is offline   Reply With Quote
Old 09-15-03, 10:06 PM   #151
Clay
Registered User
 
Clay's Avatar
 
Join Date: Oct 2002
Posts: 4,992
Default

Quote:
Originally posted by Evan Lieb
It's tough to say, though off the top of my head the [H], AT, THG, B3D, etc. would have to agree on:
What you're suggesting sounds like all of us coming up with our own set of "ISO-type standards" regarding reviews/articles/etc. If so, who is going to have final say during disputes? (which are inevitable of course) How will adjustments/updates be handled in the future? How would this all be enforced? I think these are all valid questions that need an answer before proceeding any further.

My initial reaction is that I just foresee too many chiefs and not enough indians.
Clay is offline   Reply With Quote
Old 09-15-03, 10:07 PM   #152
Joe DeFuria
Registered User
 
Join Date: Oct 2002
Posts: 236
Default

Honestly, the software developers need to be brought in on this.

Principally, if the software developer thinks it crosses the line, then that is a good indication that, well, it crosses the line.

In ANY case, "cheats" or "optimizations" need to be identified and highlighted. This will at least give the readers the opportunity to make some sort of judgement for themselves if the IHV forced trade-off is worth it.

You might consider a "slight" degredation in image quality worth a performance gain...but if the competitor has equal or close to equal performance with NO image quality degredation, the readers need to know this.

The reviewer's own judgement as to the "legitimacy" of an optimization is also welcome of course. But that should be in addition to laying out enough facts so the users can make their own judgement.

This not only helpt to keep the IHVs in check...but the reviewers as well.
Joe DeFuria is offline   Reply With Quote

Old 09-15-03, 10:12 PM   #153
Behemoth
radeon 9800 pro
 
Behemoth's Avatar
 
Join Date: Apr 2003
Location: Darkness Falls
Posts: 841
Default

this is how i see things:

there are 2 kinds of optimizations, like compression technology.
a) lossless optimization
b) lossy optimization
both are legitimate.
when you compare a) to b) on a particular benchmark without mentioning you are not comparing apple to apple to the audience, that constitutes a cheat. its a blatant cheat if you even hide the details on purpose and try to give the impression that both video cards on test are using the same settings when they are not.
Behemoth is offline   Reply With Quote
Old 09-15-03, 10:16 PM   #154
ChrisW
"I was wrong", said Chris
 
Join Date: Jul 2002
Location: standing in the corner!
Posts: 620
Default

I feel cheats are things that are done without the end user's knowledge used to deceive people. Simply turning off settings to improve performance is acceptable as long as the end user knows about it. Turning off settings while allowing the game settings to show it is still enabled is a cheat. Also, enhancing a screenshot to fool people into thinking their card is producing better quality graphics than it really is is also a cheat. Using the foreknowledge of every frame in a benchmark to increase your benchmark score (something that can't be applied to the game) is yet another example of a cheat.
__________________
AIW 9700 Pro | 1.3GHz CeleronT | 512MB PC133 SDRAM | ECS PCIPAT Mobo (Intel 815EP)
RadLinker/RadClocker
ChrisW is offline   Reply With Quote
Old 09-15-03, 10:22 PM   #155
Evan Lieb
Registered User
 
Join Date: Sep 2003
Posts: 38
Default

Quote:
Originally posted by maxpower
What you're suggesting sounds like all of us coming up with our own set of "ISO-type standards" regarding reviews/articles/etc. If so, who is going to have final say during disputes? (which are inevitable of course) How will adjustments/updates be handled in the future? How would this all be enforced? I think these are all valid questions that need an answer before proceeding any further.

My initial reaction is that I just foresee too many chiefs and not enough indians.
These are details that the web sites will pan out themselves. Readers don't need to worry about this agreement if web sites already have everything in place.

If you're wondering whether this will work, I can try and give you a general idea. I almost know for a fact that Kyle at the [H] will be more than willing to cooperate with AT in this agreement if we were to go forward with it today. AT and [H] is a huge part of the puzzle given our respective readership. I haven't met or talked with Dave of B3D, but Anand has, and from what I can gather he'd be willing to come to some sort of an agreement. Scott Damage of Tech Report is also very open minded, I wouldn't be surprised if he were as eager to come to an agreement as Kyle will be. ExtremeTech and THG are probably the biggest question marks.
Evan Lieb is offline   Reply With Quote
Old 09-15-03, 10:37 PM   #156
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Quote:
Originally posted by Evan Lieb
These are details that the web sites will pan out themselves. Readers don't need to worry about this agreement if web sites already have everything in place.

If you're wondering whether this will work, I can try and give you a general idea. I almost know for a fact that Kyle at the [H] will be more than willing to cooperate with AT in this agreement if we were to go forward with it today. AT and [H] is a huge part of the puzzle given our respective readership. I haven't met or talked with Dave of B3D, but Anand has, and from what I can gather he'd be willing to come to some sort of an agreement. Scott Damage of Tech Report is also very open minded, I wouldn't be surprised if he were as eager to come to an agreement as Kyle will be. ExtremeTech and THG are probably the biggest question marks.
Yeah, that just instills me with confidence.

"Don't worry your silly little heads with the details readers, just let us come to our own decision...I'll go talk to Kyle about it."



I thought you were serious too.
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 08:16 AM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.