3DMark2003 Review

[Greg]

Very interesting, yet unsurprising review.

http://www.aceshardware.com/read.jsp?id=60000242

[jbirney]

Originally posted by Greg
Very interesting, yet unsurprising review.

http://www.aceshardware.com/read.jsp?id=60000242

Well I have many issues with that.

3Dmarks talk about how this is a GPU and a system benchmark. If you look ONLY at 3dmark2k3 scores then his case is true. However 3Dmarks2k3 records CPUmark scores and reports that as well. Its just that those scores are not factored into the final score. However it reports those scores. If the user does not look at them then thats THEIR fault. Combine the two and you get a nice overall system indicator.

Besides furture mark is about FUTURE games not todays games. They look at:

Gun Metal = Which is a joke to be called a DX9 benchmark. It has one Vertex Shader 2 element (which should be able to run on the CPU?) but supports on PS1.1 Pixel shaders which is DX8. Doh

Unreal Tournament 2003 = Which is a DX7 game with few DX8 features sprinkeled on top.

Jedi Knight II = Horribly CPU limited game (all of the geo processing runs in software)

Comanche 4 = Probably the closes thing to a truly DX8 game test we have.

and NeverWinter Nights. Excuse me but arent these all current/old games? If you want to find out how your card does in todays games use TODAYs games. Futuremark is about games that are not out yet.


He says:

Understanding is the key to benchmarking, and hopefully this small article has provided our readers with a greater insight into 3DMark03.

It might help if they would follow their own advice and try to understand what 3dmark is in the first place.

And yes the do have a point that as an over all system becnhmark is poor.

But the pretty damming blurb on the front page is wrong.....

[Sazar]

Originally posted by jbirney
Well I have many issues with that.

3Dmarks talk about how this is a GPU and a system benchmark. If you look ONLY at 3dmark2k3 scores then his case is true. However 3Dmarks2k3 records CPUmark scores and reports that as well. Its just that those scores are not factored into the final score. However it reports those scores. If the user does not look at them then thats THEIR fault. Combine the two and you get a nice overall system indicator.

Besides furture mark is about FUTURE games not todays games. The look at:

Gun Metal = Which is a joke to be called a DX9 benchmark. It has one Vertex Shader 2 element (which should be able to run on the CPU?) but supports on PS1.1 Pixel shaders which is DX8. Doh

Unreal Tournament 2003 = Which is a DX7 game with few DX8 features sprinkeled on top.

Jedi Knight II = Horribly CPU limited game (all of the geo processing runs in software)

Comanche 4 = Probably the closes thing to a truly DX8 game test we have.

and NeverWinter Nights. Excuse me but arent these all current/old games? If you want to find out how your card does in todays games use TODAYs games. Futuremark is about games that are not out yet.


He says:



It might help if they would follow their own advice and try to understand what 3dmark is in the first place.

And yes the do have a point that as an over all system becnhmark is poor.

But the pretty damming blurb on the front page is wrong.....

we have an ongoing discussion about this very set of points on b3d...

the fact is that 3dmark03 is said to be something it is perhaps not (marketed as the gamers benchmark though lol)

of course [H] was very quick to jump on this article and hold it forth as representing their own view of things

shockingly surprising really

[Greg]

A few questions and not many answers:

o How far in the 'future' are the 3DMark2003 games set? 3DMark2001 lasted for 2 years or so, so perhaps those are like games we might play in 2004 or 5?

o Would game programmers write code and graphics programs like those in 3dm2k3? Or is that code an inefficient use of hardware. Hey, the nature scene is pretty, but full 3d grass with poor LODing? What about the rest of the world, beyond those few meters in front?

o Will future games use relatively less CPU time, or will they be like 3dm2k3, only way slower, cause they hog the CPU as well?

o How much influence did ATI have, since they bought the highest level of partnership with Futuremark? (Don't even think of responding to this please.) Eg. All you have to know is that your theoretical vertex processing is 20% better than the competition, and make a scene with simple, but high vertex count to look smarter.

o nVidia's touted strength all before and through the GeforceFX launch was 'long, complex shaders, both pixel and vertex', yet we havn't seen shaders that do anything more than be more efficient versions of DX8 level code. Perhaps developers (including Futuremark) can't think of anything to do with this feature. Related example: So we can now mix 8 textures together, 1 diffuse, 1 bump, 1 lut, 1 something, so feature goes to waste cause no one is showing off its potential. At least you could render 8 layers of 'fur' in one pass. Don't think I saw any fur in 3dm2k3.

What is a bit embarassing for our beloved friend nVidia is that the competition is regularly beating them in current AND future benchmarks. Maybe we need to see benchmarks that show off the strength of nVidia product, if that is possible. The hidden surface removal feature of the Kyro, the displacement mapping of the Matrox, and the Trueform of the ATI were great features that (to varying degrees) really didn't mean much at all.

That was quite a long rant for me, since I'm downloading some drivers and really should be going to bed. Night all.

[Nutty]

o How far in the 'future' are the 3DMark2003 games set? 3DMark2001 lasted for 2 years or so, so perhaps those are like games we might play in 2004 or 5?

They're meant to match upcoming games like Doom3 etc.. but.. see next point..

o Would game programmers write code and graphics programs like those in 3dm2k3? Or is that code an inefficient use of hardware. Hey, the nature scene is pretty, but full 3d grass with poor LODing? What about the rest of the world, beyond those few meters in front?

No. The algorithms used in 3dmark03 are not typical of what games programmers would use. Yes the benchmarks is stressing the gpu, but no it is not typical of gaming performance. Their stencil shadow algorithm is horrendous. Doom3 looks better, and I would say is around 5x faster. Their algorithms are woefully inefficient, which while do a good job of stressing the gpu, dont reflect actual game performance.

o Will future games use relatively less CPU time, or will they be like 3dm2k3, only way slower, cause they hog the CPU as well?

Future games will use more cpu than current games obviously. The more power you have, the more you can do with it. Better AI, physics etc.. The actual % of cpu power used, will probably remain quite the same tho, just as cpu's increase in power, a similar % of that power will be used.

o How much influence did ATI have, since they bought the highest level of partnership with Futuremark? (Don't even think of responding to this please.) Eg. All you have to know is that your theoretical vertex processing is 20% better than the competition, and make a scene with simple, but high vertex count to look smarter.

I dont agree with IHV's collaborating with benchmarks. A benchmark is supposed to be impartial. But in the olden days, it was quite simple. You simply drew 1000 polygons every frame, and see which got the higher frame-rate. Nowadays there are soo many things that need to be done, and soo many ways of achieving them, all with different performance attributes, so IHV's want to get in there, and steer the benchmark into showing it's hardware in the best possible way, which often ends up not being optimal for other hardware.

Games such as Doom3 that use OpenGL, have different paths to use the best aspects of each hardware. 3dmark03, and D3D dont allow this, so it doesn't represent how games like doom3 will run. As doom3 makes use of specific custom NV modes to get better performance, at the cost of some floating point accuracy that is negligable anyway.

o nVidia's touted strength all before and through the GeforceFX launch was 'long, complex shaders, both pixel and vertex', yet we havn't seen shaders that do anything more than be more efficient versions of DX8 level code. Perhaps developers (including Futuremark) can't think of anything to do with this feature. Related example: So we can now mix 8 textures together, 1 diffuse, 1 bump, 1 lut, 1 something, so feature goes to waste cause no one is showing off its potential. At least you could render 8 layers of 'fur' in one pass. Don't think I saw any fur in 3dm2k3.

Nvidia sees, as does JC (John Carmack) the imminent shift from large cpu based render farms, to gpu stations like NV30's. Soon films will use gpu's like NV30's to create films, at a fraction of the render time that cpu based render farms take.

The long shader ability is an ability to show that complex lighting algorithms used in the film industry can be done on commerical gpu hardware. Sure it's not fast enough for real-time, but thats not what companies like Pixar are interested in anyway.

The amount of time a particular shader takes is also dependant on the number of fragments (pixels) that are going to be using this shader. If you have a very small object on screen, that covers only say 100 pixels, then it is quite feasible to use very long fragment shaders on an object like this, and shorter shaders on the majority of objects/world.

There does seem to be a bit of an imagination void in benchmark stuff.. how many more times are we going to see some trees/grass, and reflective/refractive water eh?

Hardware is advancing faster than we can make use of the features, theres no doubt about that. Theres still not much stuff that uses complex DX8 style stuff.

[jbirney]

o How far in the 'future' are the 3DMark2003 games set? 3DMark2001 lasted for 2 years or so, so perhaps those are like games we might play in 2004 or 5?

I think nutty was right on with his assement. But look at ut2k3 and 3dmark2001. 3dmark2001 scores and ut2ks scores have a strong correlation (with the Kyro2 being the only exception). Both have spend a majority of time rendering DX7 stuff.

o Will future games use relatively less CPU time, or will they be like 3dm2k3, only way slower, cause they hog the CPU as well?

no one knows. Its really gonna to depend on the game. Is UT2k3 more CPU or more GPU dependent? Ahh depends doesn't. If you play off-line in AI or its "single" player mode then your pretty much CPU limited. But go on-line and frag your friends, now your in area of GPU limitations. Thus I think your gonna to see games do both. One thing is certian they will need more power

o How much influence did ATI have, since they bought the highest level of partnership with Futuremark? (Don't even think of responding to this please.) Eg. All you have to know is that your theoretical vertex processing is 20% better than the competition, and make a scene with simple, but high vertex count to look smarter.

Well good point. Dont really know. Now how about this one. NV has been a long time member at the same level of partnership with Futuremark up until late of last year. What kind of influence did they have all of these past years? How about the GF3 launch. Notice that the Apple Expo where they first showed the GF3 they also show cased 3dmark2001. Then how about the preformance analayzier that NV marketed in sold. Here they worked with Furturemark to develope tool that read 3dmark databases and recomand an NV card as an upgrade to the user.

Cold hard fact is nV has had a longer time smoozing up to benchmark authors. Look at the Good Ol Vulpine benchmark that was used a lot in the past. If you recall they had NV GF3 calls emebbed in their benchmarks. Thus when compared to the ATI cards, the ATI cards (read 8500 vrs GF3) got a much lower score. Not because it could not do the same hardware calls. But instead of using the standard OpenGL calls they use custom NV calls DroneZ was another benchmark tool that also had custom NV OpenGL calls.

They're meant to match upcoming games like Doom3 etc.. but.. see next point..

I don't think so. They only thing they tried was Doom3 like shadows which they failed at. Besides Doom3 was built around features that the orignal GF1 had according to JC. Yes it has a few DX9 stuff but by large its not a true DX9 style of game.

No. The algorithms used in 3dmark03 are not typical of what games programmers would use. Yes the benchmarks is stressing the gpu, but no it is not typical of gaming performance. Their stencil shadow algorithm is horrendous. Doom3 looks better, and I would say is around 5x faster. Their algorithms are woefully inefficient, which while do a good job of stressing the gpu, dont reflect actual game performance.

Yes we know that. However Shadows were not the only things being tested. Read PS1.1+ and PS2.0/VS2.0 also factors here.

I dont agree with IHV's collaborating with benchmarks. A benchmark is supposed to be impartial. But in the olden days, it was quite simple. You simply drew 1000 polygons every frame, and see which got the higher frame-rate. Nowadays there are soo many things that need to be done, and soo many ways of achieving them, all with different performance attributes, so IHV's want to get in there, and steer the benchmark into showing it's hardware in the best possible way, which often ends up not being optimal for other hardware.

I agree IHVs should have no part in benchmarks.

Games such as Doom3 that use OpenGL, have different paths to use the best aspects of each hardware. 3dmark03, and D3D dont allow this, so it doesn't represent how games like doom3 will run. As doom3 makes use of specific custom NV modes to get better performance, at the cost of some floating point accuracy that is negligable anyway.

I dissagree here. JC said on more than one occasion he would drop the NV30 code path if the NV3x perfromed better on the ARB2 path. Not every developer is going to want to spend the time and the money to write custom paths for their game. I think DX games will be come the norm with only a few OpenGL titles. I am not saying one is better. But take a look at what you have out today. More and more games are DX than OpenGL. Probably thanks to the Xbox.


The rest of your post I agree 100% with

[Nutty]

I don't think so. They only thing they tried was Doom3 like shadows which they failed at. Besides Doom3 was built around features that the orignal GF1 had according to JC. Yes it has a few DX9 stuff but by large its not a true DX9 style of game.

Well I think it'll be as "DX9" as alot of DX9 games. The majority of DX9 titles will merely use it for better more precise lighting, and surface algorithms, which is exactly what doom3 does on high end cards anyway.

The shadow technology (albeit double sided stenciling, UltraShadow) has been around since TNT days, its nothing new.

I dissagree here. JC said on more than one occasion he would drop the NV30 code path if the NV3x perfromed better on the ARB2 path.

I dont remember seeing that, but I think it's pretty much concluded that the ARB path will not be as fast as the NV path on the 5800, or 5900. So I doubt it will get dropped.

[jbirney]

Originally posted by Nutty
I dont remember seeing that, but I think it's pretty much concluded that the ARB path will not be as fast as the NV path on the 5800, or 5900. So I doubt it will get dropped.

Yea I remember seeing that on a slashshot or b3d interview by him. He basically say that if the NV30 perfromace was the same as the ARB2 path he would like to drop it. Its pretty said when developers have to spend their time to make optizimed paths for one set of IHVs products....

[Nutty]

To be truthful, although ppl say this, the % time Carmack spent implementing the NV30 path is negligable to the man hours spent on the whole game.

Implementing the graphics API into the engine is such a small part in the grand scheme of game making, that it isn't really an issue to experienced 3d engine writers.