Go Back   nV News Forums > Hardware Forums > Benchmarking And Overclocking

Newegg Daily Deals

Reply
 
Thread Tools
Old 06-06-03, 10:32 PM   #1
ChrisRay
Registered User
 
ChrisRay's Avatar
 
Join Date: Mar 2003
Location: Tulsa
Posts: 5,101
Default Is Hardocp trying to reform? or is this just another reviewer crusade.

I was bored. So I thought I'd post this little Forum discussion from Kyle.


I know I am not a big fan of hardocp. But I may actually read this review when it comes out, As I find "more game" reviews to be very important.


We'll see, Who knows what we'll expect, I guess when I see the review I'll know whether or not if its more trash put together or not :P



http://www.hardforum.com/showthread....5&pagenumber=1
__________________
|CPU: Intel I7 Lynnfield @ 3.0 Ghz|Mobo:Asus P7P55 WS Supercomputer |Memory:8 Gigs DDR3 1333|Video:Geforce GTX 295 Quad SLI|Monitor:Samsung Syncmaster 1680x1080 3D Vision\/Olevia 27 Inch Widescreen HDTV 1920x1080

|CPU: AMD Phenom 9600 Black Edition @ 2.5 Ghz|Mobo:Asus M3n HT Deluxe Nforce 780A|Memory: 4 gigs DDR2 800| Video: Geforce GTX 280x2 SLI

Nzone
SLI Forum Administrator

NVIDIA User Group Members receive free software and/or hardware from NVIDIA from time to time to facilitate the evaluation of NVIDIA products. However, the opinions expressed are solely those of the members
ChrisRay is offline   Reply With Quote
Old 06-07-03, 12:50 AM   #2
StealthHawk
Guest
 
Posts: n/a
Default

For all his bravado, Kyle still doesn't get it. A benchmark is a benchmark is a benchmark. Whether it is a real world gaming benchmark or a synthetic benchmark, a timedemo is just that, a timedemo. Unless the timedemo is an accurate representation of actual gameplay, it is meaningless. Not to mention that every timedemo is just as succeptible to cheating as a synthetic benchmark like 3dmark03 is.

The bottom line is that Kyle's actions have shown his unprofessionalism, his bias, and his general cluelessness. He just doesn't see the reprecussions.

The only saving grace is that Kyle realizes that the ability for reviewers to create their own timedemos easily is important. Of course again, this isn't an end all be all solution.

Some other thread comments that are relevant. On whether synthetic benchmarks are dead or not:
Quote:
You missed the point.

This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.

What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.

What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.

2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug in Splinter Cell with nvidia cards.

You missed the point.

Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.

Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on
http://www.nvnews.net/vbulletin/show...threadid=12354

On the usage of FRAPS:

Quote:
There are issues with those suggestions...which have been pointed out before.

Specifically, reviews already use more than two games. But review sites have a tendency to use the same games. IHVs will undoubtedly optimize for the games that big review sites benchmark.

As for FRAPS:
a) it is time consuming.
b) it produces varying results.
c) it renders comparing numbers between websites impossible.

C is particularly important, take Anandtech's NV35 preview for instance. By looking at other site's numbers, it is painfully clear that there is something wrong with his Quake3 4x FSAA numbers, as they show no decrease as resolution increases, while every other review site shows otherwise.
http://www.nvnews.net/vbulletin/show...&postid=130840
  Reply With Quote
Old 06-07-03, 08:13 AM   #3
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Unhappy Kyle is a tool.

Which makes me sad, but only 'cause of Brent.

If this is the straw that breaks the camels back and brings nVidia the national attention in the news that they seem to be trying to get THEN I think you might see [t]ardOCP mysteriously "reform" and "discover" the errors of it's ways...but not until then.

Anyone wanna take bets on how long Brent will last there? I'm guessing he'll quit before the month is out.
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Old 06-07-03, 10:10 AM   #4
Kruno
TypeDef's assistant
 
Join Date: Jul 2002
Location: Australia
Posts: 1,641
Send a message via ICQ to Kruno Send a message via AIM to Kruno
Default

As for FRAPS:
Quote:
a) it is time consuming.
-how so? AFAIK I enter the program and then go into the game and take shots with num lock
b) it produces varying results.
-I have been using FRAPS for years and I have benched games over and over using the same settings my results had an error ratio so low that I have trouble computing it to the exact fento
c) it renders comparing numbers between websites impossible.
No comment...
__________________
"Never before has any voice dared to utter the words of that tongue in Imladris, Mr. Anderson" - Elrond LOTR
Kruno is offline   Reply With Quote
Old 06-07-03, 10:26 AM   #5
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Do you have any IDEA how long it takes some of us to add up all the numbers and count 'em and divide it to find out our averages?!?
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote
Old 06-07-03, 10:52 AM   #6
ragejg
nV News Alumni
 
ragejg's Avatar
 
Join Date: Sep 2002
Location: Finger Lakes of Hell, NY
Posts: 5,399
Send a message via AIM to ragejg
Default ...

it is hard to get the real world/synth bench ratio correct, isn't it?

Even though it is time consuming, I personally am gonna use fraps a little more lately...
__________________

2010-2011 Reviews: GTX 570 | GTX 580 | GTS 450 | GTX 460 | GTX 465
Pre-2010 Reviews: 6600 GT | XMS 4400 DDR | SilenX Cooler | 6800 | 5900 XT | Personal Cinema | 5900 NU


______________________________________________
Phenom II x6 1090t @ 4.0 ghz | Asus M498TD-EVO Am3 SLI nForce 980a | 2x EVGA GTX 560 SLI | 2x4gb DDR3-1333 | Antec EarthWatts EA650 PSU | 60gb Mushkin Calisto Enhanced Sandforce SSD | 2x WD2500KS RAID 0 | Sunbeam Tuniq 3 case | 24" Asus 19x10 LED LCD | 26" Panasonic 720p TV | Sidewinder X5 mouse | Logitech MX5500 & Revolution mouse | Altec Lansing 5.1 THX-Certified audio | Win 7 Ultimate | desk | couch


ragejg is offline   Reply With Quote
Old 06-07-03, 08:11 PM   #7
StealthHawk
Guest
 
Posts: n/a
Default

Quote:
Originally posted by K.I.L.E.R
As for FRAPS:
-how so? AFAIK I enter the program and then go into the game and take shots with num lock
Because you have to sit there and play the game?

Quote:
-I have been using FRAPS for years and I have benched games over and over using the same settings my results had an error ratio so low that I have trouble computing it to the exact fento
Because you have to go through a similar "run" to get the same results. Playing on another level or a different area in the same level can drastically change the results.

Quote:
No comment...
As proven with Anand's numbers, we need to be able to compare review sites to see if their numbers are reasonable or not.


When reviewers start using specialized tests it means we:
a) have to trust their results as being correct
b) have to trust that they aren't skewing their results, finding scenes in which one IHV excells, etc.

edit: for clarity
  Reply With Quote
Old 06-07-03, 08:34 PM   #8
digitalwanderer
 
digitalwanderer's Avatar
 
Join Date: Jul 2002
Location: Highland, IN USA
Posts: 4,944
Default

Quote:
Originally posted by StealthHawk
Because you have to go through a similar "run" to get the same results. Playing on another level can drastically change the results.
Well, yeah.

I've found that recording my own demo and then using fraps on it and only changing settings/cards seems to work quite accurately.
__________________
[SIZE=1][I]"It was very important to us that NVIDIA did not know exactly where to aim. As a result they seem to have over-engineered in some aspects creating a power-hungry monster which is going to be very expensive for them to manufacture. We have a beautifully balanced piece of hardware that beats them on pure performance, cost, scalability, future mobile relevance, etc. That's all because they didn't know what to aim at."
-R.Huddy[/I] [/SIZE]
digitalwanderer is offline   Reply With Quote

Old 06-07-03, 10:44 PM   #9
ChrisRay
Registered User
 
ChrisRay's Avatar
 
Join Date: Mar 2003
Location: Tulsa
Posts: 5,101
Default

It's very difficult for me to test Kyles motives, I dunno if he has good intentions and just comes out like a moron. (He and I do agree on 3dmark being useless however)

But It's hard to answer these questions


What is Kyles motivations?

Is Kyle a sellout?

Is Kyle trying but really failing in the delivery?

Does he truly believe so strongly about this that he thinks he can just change over night how benches are done.


I wish I could form a specific opinion about him. But he changes his rational as quick the wind changes, So its hard to respect the guy currently
__________________
|CPU: Intel I7 Lynnfield @ 3.0 Ghz|Mobo:Asus P7P55 WS Supercomputer |Memory:8 Gigs DDR3 1333|Video:Geforce GTX 295 Quad SLI|Monitor:Samsung Syncmaster 1680x1080 3D Vision\/Olevia 27 Inch Widescreen HDTV 1920x1080

|CPU: AMD Phenom 9600 Black Edition @ 2.5 Ghz|Mobo:Asus M3n HT Deluxe Nforce 780A|Memory: 4 gigs DDR2 800| Video: Geforce GTX 280x2 SLI

Nzone
SLI Forum Administrator

NVIDIA User Group Members receive free software and/or hardware from NVIDIA from time to time to facilitate the evaluation of NVIDIA products. However, the opinions expressed are solely those of the members
ChrisRay is offline   Reply With Quote
Old 06-07-03, 11:17 PM   #10
Kruno
TypeDef's assistant
 
Join Date: Jul 2002
Location: Australia
Posts: 1,641
Send a message via ICQ to Kruno Send a message via AIM to Kruno
Default

SH, you can trust me.
__________________
"Never before has any voice dared to utter the words of that tongue in Imladris, Mr. Anderson" - Elrond LOTR
Kruno is offline   Reply With Quote
Old 06-07-03, 11:45 PM   #11
ChrisRay
Registered User
 
ChrisRay's Avatar
 
Join Date: Mar 2003
Location: Tulsa
Posts: 5,101
Default

Quote:
Originally posted by K.I.L.E.R
SH, you can trust me.

You always said that.. then you left me on my old forum.. And then I came here.. finding you with others. Our relationship will never be the same again
__________________
|CPU: Intel I7 Lynnfield @ 3.0 Ghz|Mobo:Asus P7P55 WS Supercomputer |Memory:8 Gigs DDR3 1333|Video:Geforce GTX 295 Quad SLI|Monitor:Samsung Syncmaster 1680x1080 3D Vision\/Olevia 27 Inch Widescreen HDTV 1920x1080

|CPU: AMD Phenom 9600 Black Edition @ 2.5 Ghz|Mobo:Asus M3n HT Deluxe Nforce 780A|Memory: 4 gigs DDR2 800| Video: Geforce GTX 280x2 SLI

Nzone
SLI Forum Administrator

NVIDIA User Group Members receive free software and/or hardware from NVIDIA from time to time to facilitate the evaluation of NVIDIA products. However, the opinions expressed are solely those of the members
ChrisRay is offline   Reply With Quote
Old 06-08-03, 05:19 AM   #12
StealthHawk
Guest
 
Posts: n/a
Default

Quote:
Originally posted by ChrisRay
It's very difficult for me to test Kyles motives, I dunno if he has good intentions and just comes out like a moron. (He and I do agree on 3dmark being useless however)

But It's hard to answer these questions


What is Kyles motivations?
He wants hits on his website and will do anything to get them.

Quote:
Is Kyle a sellout?
I think he is.

Quote:
Is Kyle trying but really failing in the delivery?
He either isn't trying, or he's really an idiot. Everything he has said about all these 3dmark03 issues has been mind numbingly stupid. If he actually believes half the stuff he says....

Quote:
Does he truly believe so strongly about this that he thinks he can just change over night how benches are done.
Did he actually say that? It looks like he has been trying to change the way benchmarks are done for a few months. You gotta start somewhere. For that, I applaud his actions.
  Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 08:37 AM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.