New [H] Editorial on Benchmarking

[jbirney]

Originally posted by bkswaney
Yes but AM3 is based off a real game. 3DM03 is not.
Or have I been lead a stray and AM3 is a sen?
Is it not based from the Krass engine?

Ok So you want benchmarks based off real engines. Riddle me this:

1) Kyro2 gets is butt handed to it in 3Dmark2001 by the GF2 Mx.
2) In Max Pane, the kyro2 usally had more than x2 FPS than the same GF2 mx
3) 3Dmark2001 and MaxPane used the SAME ENGINE!

Doh!

Besides look at UT2003 FPS scores and Splinter Cell FPS scores. Both based off the same Unreal Engine so both should have the same score right? Nope not even close. So if two games based off the same engine have different results then does the engine really matter? Nope.

I agree with you. Games matter. However dont ever underestimate the power of 3dmarks. Large OEMs and other big players base a lot of money on how cards score in benchmarks including 3dmark. I dont agree with them but thats what they do. If 3dmark was such a non-issue then why does on IHV keep on optimizing it using 3 different methods?

[oqvist]

Well the FX series so far is a totally lost generation. Can´t believe nVidia screwed up so bad with the FX series that they felt forced to pull all the moves they have done with it.

I will try cover my ear and be as objective I can when the NV40 comes out and hope that they don´t continue the same route with that as they did with the previous FX cards...

Or else just continuing buying ATI cards and forget that nVidia never existed...

And about ATI:s compilers. You can remove this optimizations with 2 mouseclicks in the control menu.

Whereas nVidia driver engineers can´t manage doing the same with the FX series because they are so complex I guess...

[Sazar]

meh one PR guys posting to contradict another :(

anyways I wonder if kyle is going to change his editorial in light of the retraction by nvidia (apparently) that their compiler is in fact NOT messed with

:cool:

its getting curiouser and curiouser (sry... can't remmeber what movie thats from)

[legion88]

Originally posted by euan
I'm confused.

What is the difference between a synthetic test that does many various 3d operations (geometery, textures, shaders), and say a game fly-by or recorded demo?

In an ideal world, there would be no difference. But we don't live in an ideal world.

A properly designed test for graphics cards would not have any vendor-specific code. That is, the test won't have subroutines like "if card = ATI, then run routine X. if card = NVDIA, then run routine Y'. All the cards would use the same routines so all the cards would be treated the same by the test.

In games, however, that is not the case. Back in the old days of 3dfx, game developers loved glide for more reasons that it being simple to use. Game developers can be very confident that their glide code would run on any 3dfx-based card from the various video card manufacturers because they all are using the same glide drivers.

For Direct3D and OpenGl, that is not the case. A developer's OGL code is not guaranteed to run well on every OGL-capable card because the various cards are not using the same OGL drivers. A developer's D3D code is not guaranteed to run well on every D3D-capable card because the various cards are not using the same D3D drivers. It is not uncommon to see a D3D game crash often using one card while not crash at all on another.

By necessity, the developers had to implement vendor-specific code in their programs for performance issues or to prevent "show-stopping" bugs.

So for game benchmarks, video card performance is not the only thing being measured. The game benchmarks also measure how well the developer can fine-tune their game code for specific vendors.

A properly designed "synthetic" benchmark don't have vendor specific code. All the cards are treated the same.

[Malfunction]

I agree with what you said legion88, although I can't shake this feeling that what everyone was worried about is going to happen the more I read about the latest DX9 games being *delayed... I mean coming out. ;)

Videocard Industry/Game Dev becoming more like console market:

At first, I was a little reserved as to that being a good idea. Now I kinda am happy about it. I think it might make features that are not normally used like Tru-Form (like sxotty said in another post) start to be implemented. More like the more features a card has and are used while providing terrific performance will be the victor. :) Why is that such a bad thing?

Reminds me of the console market of the past. One game I can recall I had on both the PS1 and Sega Saturn was Resident Evil. (this is my example so bare with me ;) ) While the Saturn was *supposedly inferior to the PS1, I still think that a few game(s) (Resident Evil in particular) was better looking and played better on the Saturn.

Director's cut on the PS1 version ment jack because I still think it looked better on the Saturn.

When they began weighing in the XBox against the competition, first thing they pointed out was game play on each system with the same title. I think there is enough room for Dev's to showcase for either IHV, how they go about it is what I am curious.

I would however enjoy seeing alot of the features with ATi that are not implemented in game(s) to be showcased soon. In my eyes, all this would do is benefit ATi because their tech is superior at the moment which would only force Nvidia to get on the ball.

My $.02...

Peace,

:)

[NickSpolec]

Unfortunately, Saturn wasn't supposed to be a 3D machine. Sega originally planned for it to be a power house 2D system (in fact, a home version of their System32 arcade hardware), with just enough 3D power to do Model 1 games.

But then Sony showed off PSX, and Sega rushed back to development. They added, in theory, more 3D functionality then PSX had. But, it was really a mess in conventional terms. If you had a good handle on the system (like Yu Suzuki), you could create pretty impressive stuff. But the common developer found the 3D programming far to complex and time consuming, and the results lack-luster compared to the ease of PSX.

[Malfunction]

Originally posted by NickSpolec
Unfortunately, Saturn wasn't supposed to be a 3D machine. Sega originally planned for it to be a power house 2D system (in fact, a home version of their System32 arcade hardware), with just enough 3D power to do Model 1 games.

But then Sony showed off PSX, and Sega rushed back to development. They added, in theory, more 3D functionality then PSX had. But, it was really a mess in conventional terms. If you had a good handle on the system (like Yu Suzuki), you could create pretty impressive stuff. But the common developer found the 3D programming far to complex and time consuming, and the results lack-luster compared to the ease of PSX.

Ok, Alot of the Dreamcast games looked better than the PS2. Soul Caliber is one I can think of off the top of my head. Won't deny that Sony made it easier for Dev's to code, that is for certain. Hoever they have not always had the best presentation of a game title by allowing it either. Double edge sword I suppose...

Peace,

:)

[ChrisRay]

Originally posted by Malfunction
Ok, Alot of the Dreamcast games looked better than the PS2. Soul Caliber is one I can think of off the top of my head. Won't deny that Sony made it easier for Dev's to code, that is for certain. Hoever they have not always had the best presentation of a game title by allowing it either. Double edge sword I suppose...

Peace,

:)


Heh Played Grandia 2 on the Dreamcast then played it on the PS2? It looks awful. The Dreamcast would at least anti alias the game...


Speaking of grandia 2. I love the PC version :)

[NickSpolec]

Ok, Alot of the Dreamcast games looked better than the PS2. Soul Caliber is one I can think of off the top of my head. Won't deny that Sony made it easier for Dev's to code, that is for certain. Hoever they have not always had the best presentation of a game title by allowing it either. Double edge sword I suppose...


Of course, Dreamcast had some serious untapped force. I would estimate that no game for DC consumed more then 75-80 percent of the total Dreamcast power. Given an extra year of Sega developed games, I'm sure we would have gotten close to the 100 percent mark.

Heh Played Grandia 2 on the Dreamcast then played it on the PS2? It looks awful. The Dreamcast would at least anti alias the game...


Contrary to popular belief, Dreamcast used no anti-aliasing.

The reason why no one really heard of jaggies (in the console market), or noticed them before Playstation 2 hit, was not because of anti-aliasng on Dreamcast, but the Dreamcast's flicker filter.

You should know about flicker filter, if you use TV out on video cards. The default Dreamcast flicker filter was equal to a 2 on GeForce hardware (when using the control panel), on ATI hardware... Meh, one or two notches below full.

Playstation 2 used no flicker filter (not in the first 6 months or so), which is why all the edges were so hard on all the games, while on Dreamcast the edges were quite soft. In fact, you can still find PS2 games that still use the default setting on 0 (zero) on flicker filter. Dreamcast, from the beginning, used the equivalent of 2 (and no game uses a different setting, that I know of --- simply because the default setting was effective [if not a little blurry])

[Socos]

Originally posted by ginfest
HB, I agree with some of what you are saying but have to question the above:
I have a 5900 and a 9800 and yes the 9800 runs Max Payne and my other games very well. I run all my games at 1280x960 4xAA/8xAF. I had to put the 5900 back in to get Call of Duty running and have since continued playing Max Payne and haven't changed any settings. I run it with the console enabled to get FPS showing and haven't noticed a big diff. I don't have and exact numbers but both cards run it at the above settings at 60 FPS or better.
Anyway, if you're talking IQ, I'm not sitting here while playing the game and other games that I play saying "...s**t, that looks crappy compared to my 9800.." and so on. Yes I know that comparing individual frames will show better AA for the 9800 but I'm talking game experience. And no I'm not saying that it doesn't count if you can't see it, just that the difference is not enough to ruin the game for me.
I suppose you could say it's just me-I wonder if others a have run both cards recently and saw a noticeable difference, ie something that makes you say "WTF, I can't play this game like this.."?

My $0.02

Mike G

So what you are saying is both cards play the games you play? Or Call of Duty does not work on a 9800? You lost me on that.

And your probably right most of the arguments made are BS.. Basically it boils down to both cards are capable of playing all of todays games at decent frame rates with decent IQ.

I'm still gonna get a 9800XT!!

[Socos]

Originally posted by cthellis
And each game represents its OWN ruleset... not ANY OTHER GAME'S ruleset. Their objectives are totally different.

Synthetics are useful so long as you recognize the context. Most synthetics try to concentrate on generalities, and BECAUSE they do so they will be a more useful predictor than looking at a single game--or even a small set of games. Game benches don't even necessarily match up with actual gameplay situations either, so the way you measure performance may be lying to you as well. (Not to mention IHV optimizations may affect their benches but not carry over as much or at ALL to general gameplay.)

Everything has it's place and can be used properly so long as you KNOW what its place is. Nothing is by nature "worthless"--its worth just has to be understood and put in the proper light.

But certainly no one should make an unearthly deal over ONE number--not one synthetic bench, nor one game performance number. (Nor one resolution, nor one quality setting...) That habit is stupid no matter what you're talking about.

But you get down to this: Does the Synthetic truly represent actually gameplay results. And I think in the case of 3Dmark the answer is yes. TAOD, preliminary Hl2, and others have shown that PS2.0 doesn't delivery on NV35. I am not sure if the percent difference is the same on TRAOD vs 3Dmark, but it is a sizeable difference.

[cthellis]

Originally posted by Ruined
I hope 3dmark keeps changing their code. Means more frequent driver updates for Nvidia users! :afro:

<laughs> Indeed. Silver lining, what what? ;)

Originally posted by Malfunction
Videocard Industry/Game Dev becoming more like console market:


Yes, and I think this is decidedly a BAD thing. PC's shouldn't be borrowing the worse elements of consoling, and consoles certainly shouldn't be seeking to throw in the downsides of PC gaming either. Convergence will, of course, ALWAYS be happening--but they should certainly be going about it better.

Originally posted by Malfunction
Ok, Alot of the Dreamcast games looked better than the PS2. Soul Caliber is one I can think of off the top of my head.

You should probably defrag the top of your head, then. ;) Soul Calibur never appeared on any system but the Dreamcast. Soul Calibur II is a different matter, but that's a generational skip. You might be thinking of Dreamcast's other notable fighter--DOA2--which had a PS2 version in DOA2: Hardcore.

[NickSpolec]

I think he meant to say that "Alot of Dreamcast games look better then (some) PS2 games."