Is Hardocp trying to reform? or is this just another reviewer crusade.

[ChrisRay]

I was bored. So I thought I'd post this little Forum discussion from Kyle.


I know I am not a big fan of hardocp. But I may actually read this review when it comes out, As I find "more game" reviews to be very important.


We'll see, Who knows what we'll expect, I guess when I see the review I'll know whether or not if its more trash put together or not :P



http://www.hardforum.com/showthread....5&pagenumber=1

For all his bravado, Kyle still doesn't get it. A benchmark is a benchmark is a benchmark. Whether it is a real world gaming benchmark or a synthetic benchmark, a timedemo is just that, a timedemo. Unless the timedemo is an accurate representation of actual gameplay, it is meaningless. Not to mention that every timedemo is just as succeptible to cheating as a synthetic benchmark like 3dmark03 is.

The bottom line is that Kyle's actions have shown his unprofessionalism, his bias, and his general cluelessness. He just doesn't see the reprecussions.

The only saving grace is that Kyle realizes that the ability for reviewers to create their own timedemos easily is important. Of course again, this isn't an end all be all solution.

Some other thread comments that are relevant. On whether synthetic benchmarks are dead or not:

You missed the point.

This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.

What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.

What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.

2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug in Splinter Cell with nvidia cards.

You missed the point.

Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.

Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on

http://www.nvnews.net/vbulletin/show...threadid=12354

On the usage of FRAPS:

There are issues with those suggestions...which have been pointed out before.

Specifically, reviews already use more than two games. But review sites have a tendency to use the same games. IHVs will undoubtedly optimize for the games that big review sites benchmark.

As for FRAPS:
a) it is time consuming.
b) it produces varying results.
c) it renders comparing numbers between websites impossible.

C is particularly important, take Anandtech's NV35 preview for instance. By looking at other site's numbers, it is painfully clear that there is something wrong with his Quake3 4x FSAA numbers, as they show no decrease as resolution increases, while every other review site shows otherwise.

http://www.nvnews.net/vbulletin/show...&postid=130840

[digitalwanderer]

Which makes me sad, but only 'cause of Brent.

If this is the straw that breaks the camels back and brings nVidia the national attention in the news that they seem to be trying to get THEN I think you might see [t]ardOCP mysteriously "reform" and "discover" the errors of it's ways...but not until then.

Anyone wanna take bets on how long Brent will last there? I'm guessing he'll quit before the month is out.

[Kruno]

As for FRAPS:

a) it is time consuming.
-how so? AFAIK I enter the program and then go into the game and take shots with num lock
b) it produces varying results.
-I have been using FRAPS for years and I have benched games over and over using the same settings my results had an error ratio so low that I have trouble computing it to the exact fento
c) it renders comparing numbers between websites impossible.
No comment...

[digitalwanderer]

Do you have any IDEA how long it takes some of us to add up all the numbers and count 'em and divide it to find out our averages?!?

[ragejg]

it is hard to get the real world/synth bench ratio correct, isn't it?

Even though it is time consuming, I personally am gonna use fraps a little more lately...

Originally posted by K.I.L.E.R
As for FRAPS:
-how so? AFAIK I enter the program and then go into the game and take shots with num lock

Because you have to sit there and play the game?

-I have been using FRAPS for years and I have benched games over and over using the same settings my results had an error ratio so low that I have trouble computing it to the exact fento

Because you have to go through a similar "run" to get the same results. Playing on another level or a different area in the same level can drastically change the results.

No comment...

As proven with Anand's numbers, we need to be able to compare review sites to see if their numbers are reasonable or not.


When reviewers start using specialized tests it means we:
a) have to trust their results as being correct
b) have to trust that they aren't skewing their results, finding scenes in which one IHV excells, etc.

edit: for clarity

[digitalwanderer]

Originally posted by StealthHawk
Because you have to go through a similar "run" to get the same results. Playing on another level can drastically change the results.

Well, yeah.

I've found that recording my own demo and then using fraps on it and only changing settings/cards seems to work quite accurately.

[ChrisRay]

It's very difficult for me to test Kyles motives, I dunno if he has good intentions and just comes out like a moron. (He and I do agree on 3dmark being useless however)

But It's hard to answer these questions


What is Kyles motivations?

Is Kyle a sellout?

Is Kyle trying but really failing in the delivery?

Does he truly believe so strongly about this that he thinks he can just change over night how benches are done.


I wish I could form a specific opinion about him. But he changes his rational as quick the wind changes, So its hard to respect the guy currently

[Kruno]

SH, you can trust me.

[ChrisRay]

Originally posted by K.I.L.E.R
SH, you can trust me.

You always said that.. then you left me on my old forum.. And then I came here.. finding you with others. Our relationship will never be the same again

Originally posted by ChrisRay
It's very difficult for me to test Kyles motives, I dunno if he has good intentions and just comes out like a moron. (He and I do agree on 3dmark being useless however)

But It's hard to answer these questions


What is Kyles motivations?

He wants hits on his website and will do anything to get them.

Is Kyle a sellout?

I think he is.

Is Kyle trying but really failing in the delivery?

He either isn't trying, or he's really an idiot. Everything he has said about all these 3dmark03 issues has been mind numbingly stupid. If he actually believes half the stuff he says....

Does he truly believe so strongly about this that he thinks he can just change over night how benches are done.

Did he actually say that? It looks like he has been trying to change the way benchmarks are done for a few months. You gotta start somewhere. For that, I applaud his actions.