For all his bravado, Kyle still doesn't get it. A benchmark is a benchmark is a benchmark. Whether it is a real world gaming benchmark or a synthetic benchmark, a timedemo is just that, a timedemo. Unless the timedemo is an accurate representation of actual gameplay, it is meaningless. Not to mention that every timedemo is just as succeptible to cheating as a synthetic benchmark like 3dmark03 is.
The bottom line is that Kyle's actions have shown his unprofessionalism, his bias, and his general cluelessness. He just doesn't see the reprecussions.
The only saving grace is that Kyle realizes that the ability for reviewers to create their own timedemos easily is important. Of course again, this isn't an end all be all solution.
Some other thread comments that are relevant. On whether synthetic benchmarks are dead or not:
You missed the point.
This changes everything. We now know how nvidia cheated. Two specific problems are inserting clipping planes and changing shader code.
What this means to us as gamers:
1) Any timedemo is now suspect because of possible clipping planes. This includes all those nifty timedemos that are readily available in games that people and reviewers like to run. I'm pretty sure it would be possible to have the driver detect when you are going off the rail, so this means that any IHV could cheat in the timedemo and inflate timedemo scores without being caught easily. This cheat would not increase scores during actual gameplay at all.
What this means is that reviewers will either have to use their own personal demos that will not be distributed to the public, or will have to run FRAPS. There are obvious disadvantages to both of these methods. Both suffer from the inability for gamers to compare their systems with reviews. The FRAPS approach obviously takes a great deal of time, and results will not be consistent between runs.
2) Any game or benchmark with shaders is now suspect. Who knows when and where shader code is being re-written to decrease quality and increase speed like nvidia did with 3dmark03? There is either cheating or a bug in Splinter Cell with nvidia cards.
You missed the point.
Benchmarking as we know it will have to change, and change drastically, to help cut down on cheats. Synthetic benchmarks aren't the only things in trouble...all timedemos, which are basically the only method of benchmarking in use by reviews today, are in jeopardy.
Does this mean that there is cheating going on in all timedemos, from games to synthetic benchmarks alike? No. But it means whenever we as consumers see huge increases in performance from new drivers that something fishy might be going on
On the usage of FRAPS:
There are issues with those suggestions...which have been pointed out before.
Specifically, reviews already use more than two games. But review sites have a tendency to use the same games. IHVs will undoubtedly optimize for the games that big review sites benchmark.
As for FRAPS:
a) it is time consuming.
b) it produces varying results.
c) it renders comparing numbers between websites impossible.
C is particularly important, take Anandtech's NV35 preview for instance. By looking at other site's numbers, it is painfully clear that there is something wrong with his Quake3 4x FSAA numbers, as they show no decrease as resolution increases, while every other review site shows otherwise.