Mike's 3DMark Commentary

[jbirney]

Mike,

I agree with your ideas and beliefs on why 3dmarks is useful and why its not. However I don't really care for nV's change of heart on this issue. As you stated 3dmark has never been a good gauge for judging how well cards do in real games. But ask your self has it ever? Did it at some point fall from being super accurate to what it is today?

No it never has, never was and probably never will be. However in the past nV has never taken this stance. They as well as we knew that it was not the "gamers benchmark" but why did they not say something back for 3dmark2000? 20001? Why now? What really burns my bacon with their stance is that nV actively marketed and pushed their Performance analyzer which only went out an read the Orb data base and reported back which nV card the users should get. Then the sold this to companies like Westwood so they use that to recommend nV cards in their games (C&C:Renagade for example). Why is nV all of a sudden worried about the gamers benchmark?

I am sorry Mike but I wont cut them slack on this department. nV has always backed 3dmarks scores and now when the tech in 3dmark does not benefit them (for example 3dmark2003 using PS1.4 which nV dose not have native hardware support for), they get all hissy.

[Uttar]

My understanding of the situation is amazingly simple. And if I'm right, this decision makes a LOT of sense.

1. nVidia driver team is *very* busy ATM. They're trying to optimize the NV30, to work on NV34 software shader routines and to optimize NV31 because it's going to be announced soon. Thus, if they can gain time by not worrying too much about 3DMark 2003 optimizations, they might actually get the other things done in time.

2. nVidia expects ATI to have put a lot of time on optimizing their 3DMark 2003 score. They'd be delighted if they could have made them lose their time.

3. 3DMark 2003 use few *different* shader programs. The NV30 is actually faster than the R300 when a lot of shader programs are used, because they're stored in video memory. So they obviously don't like that.

4. 3DMark 2003 uses VS2.0. and PS2.0., but *all* of those shaders are executed in one pass on the R300 AFAIK. That means nVidia isn't getting any advantage from their huge NV30 PS/VS flexibility. That sure doesn't make them happy...

Personally, I really don't think nVidia even cares to the slightest degree about PS1.4.
nVidia doesn't care about the NV2x anymore. What they want, now, is a huge NV3x line-up: everything from $99 to $499 will soon be NV3x. So that's really not their problem...


Uttar

[Typedef Enum]

Let me put it this way...Almost everybody can understand nVidia's point. Hell, there have been a LOT of people expressing those exact same sentiments for years! Here's the problem...

The problem is that, all of a sudden, nVidia doesn't want to play by the rules. You can take virtually any of nVidia's key complaints about 3DMark2003, and apply it to 3DMarkxx. Why? It's utterly clear why. Because they are no longer THE high-performance leader, and they won't be for some time.

It's spoiled milk, and we all know it. So, when nVidia is no longer able to claim the performance crown, they decide to withdraw from the 3DMark team, and complain. Never voiced those concerns at all when 3DMark 99and 3DMark 2000 were doing similar tests. It's just totally lame, and we all know it.

[|JuiceZ|]

Originally posted by Typedef Enum
It's spoiled milk, and we all know it. So, when nVidia is no longer able to claim the performance crown, they decide to withdraw from the 3DMark team, and complain. Never voiced those concerns at all when 3DMark 99and 3DMark 2000 were doing similar tests. It's just totally lame, and we all know it.

Its really sad, but I have to agree :( That point clearly out weighs the preexistent statements nvidia is making now.

[Solomon]

I totally agree with Typedef Enum,

Hell I been preaching all the time that 3DMark is just a pissing contest, nothing more nothing less. Using the games you play is what should be used in testing frame rate, etc... I've heard people say using UT2k3 is stupid. It's so CPU limited. Even if it is. It's a game alot of people play. So even if it's CPU limited I rather see those results then a synthetic benchmark no one plays.

Typedef Enum said it perfectly. This just makes Nvidia look like whiners. Now that they aren't the performance leaders they are crying this and that.

Regards,
D. Solomon Jr.
*********.com

[Uttar]

As I said, I think there *is* a new element: drivers.
In the past, nVidia always had a lot of time to optimize 3DMark. Today, they don't because they got their driver team already working on a LOT of things at once.

Does this make nVidia sudden refusal of 3DMark 2003 more acceptable? IMO: Yes, a little more. But it still isn't perfectly acceptable.

Uttar

[ragejg]

Their opinion will probably shift gradually with time, as their product gradient fills out...

I think their standing was sort of a knee jerk reaction, cuz everyone's so hungry right now, and conclusions are being drawn faster than nvidia can come up with answers.

[Typedef Enum]

Let me also add one more thing...

It's utterly insulting, IMHO, to see nVidia employees say things to the effect of...

"We would be much better off optimizing our drivers for actual games, rather than synthetic benchmarks."

I don't know who these guys are fooling, but it's certainly not me. nVidia rised to the top by doing that EXACT same thing...what the heck are they talking about? They have been tweaking their own drivers to sport significantly high Quake/3DMark numbers for years, and then using that data to demonstrate just how much performance their engineers could eek out of their drivers...

and then the PR guys would say, "This driver will yield some 40% increase."

Anyhow, this part of their response is the most laughable, IMHO. I mean, talk about the pot calling the kettle black.

[Ratchet]

I don't know what the sudden backlash against 3DMark is all about from people (nVidia, nVnews, and [H] most notably). It's not like reviewers only focus on 3DMark scores, most reviews I read have all types of different games from different genres using the two main APIs (DX and OGL). Frankly, any review that states only 3DMark numbers isn't worth reading.

Even the most hardcore nVidia fan can see that this is nothing but an nV temper tantrum on par with what you'd expect from a child who can't get their own way. Is this immature attitude supposed to be part of nVidias "new marketing" approach?

[StealthHawk]

Originally posted by Ratchet
I don't know what the sudden backlash against 3DMark is all about from people (nVidia, nVnews, and [H] most notably). It's not like reviewers only focus on 3DMark scores, most reviews I read have all types of different games from different genres using the two main APIs (DX and OGL). Frankly, any review that states only 3DMark numbers isn't worth reading.

most good reviews do benchmark several games and not just 3dmark. but there are lots of "reviews" that consist of nothing other than maybe your generic Quake3 score, 3dmark score, and maybe one other DX game score.

obviously not true of any of the major sites, but was and is still true of many of the smaller sites, magazines, etc.

[Solomon]

Isn't this sort of ironic. Considering ATi and their Quake3.exe debacle in the drivers. How everyone harped on them for this cheating bit. Now it seems that people are coming out of the wood work saying that Nvidia did indeed optimize drivers for the synthetic benchmark when before all of this no one really wanted to say such words. Now it seems people can't get enough of saying that Nvidia is mad because their drivers aren't optimized or what not for a specific benchmark.

It's weird. First Nvidia users wouldn't admit that Nvidia was indeed doing that. Now that this has happened you can't get enough of them saying that they are mad because their optimizations aren't happening right or what not.

It's like the Twilight Zone! :eek:

Regards,
D. Solomon Jr.
*********.com

[saturnotaku]

Originally posted by Solomon
Hell I been preaching all the time that 3DMark is just a pissing contest, nothing more nothing less. Using the games you play is what should be used in testing frame rate, etc... I've heard people say using UT2k3 is stupid. It's so CPU limited. Even if it is. It's a game alot of people play. So even if it's CPU limited I rather see those results then a synthetic benchmark no one plays.


And what have I been saying since practically the first day I joined these forums?

About the the only thing I think 3dmark is good for is stressing the graphics subsystems to test overclocking. Anything else is pittance. I bought my video card to play games and play games is what I do.

[|JuiceZ|]

Originally posted by saturnotaku
And what have I been saying since practically the first day I joined these forums?

About the the only thing I think 3dmark is good for is stressing the graphics subsystems to test overclocking. Anything else is pittance. I bought my video card to play games and play games is what I do.

right on sat, I too only use it for testing stability after overclocking and have always been a fan of realworld benchmarking.

[Rogozhin]

"3DMark03 combines custom artwork with a custom rendering engine that creates a set of demo scenes that, while pretty, have very little to do with actual games. It is much better termed a demo than a benchmark. The examples included in this report illustrate that 3DMark03 does not represent games, can never be used as a stand-in for games, and should not be used as a gamers’ benchmark."

NVIDIA:
"Unfortunately, Futuremark chose a flight simulation scene for this test (game 1). This genre of games is not only a small fraction of the game market (approximately 1%), but utilizes a simplistic rendering style common to this genre. Further, the specific scene chosen is a high altitude flight simulation, which is indicative of only a small fraction of that 1%."

"For all intents and purposes game tests 2 and 3 are the same test. They use the same rendering paths and the same feature set. The sole difference in these tests appears to be the artwork. This fact alone raises some questions about breadth of game genres addressed by 3DMark03. --- These two tests attempt to duplicate the “Z-first” rendering style used in the upcoming first-person shooter game, “Doom 3”. They have a “Doom-like” look, but use a bizarre rendering method that is far from Doom 3 or any other known game application."

"Finally, the choice of pixel shaders in game tests 2 and 3 is also odd. These tests use ps1.4 for all the pixel shaders in the scenes. Fallback versions of the pixel shaders are provided in ps1.1 for hardware that doesn’t support ps1.4. Conspicuously absent from these scenes, however, is any ps1.3 pixel shaders. Current DirectX 8.0 (DX8) games, such as Tiger Woods and Unreal Tournament 2003, all use ps1.1 and ps1.3 pixel shaders. Few, if any, are using ps1.4."

"This year’s 3DMark has a new nature scene (game 4). It is intended to represent the new DirectX 9.0 (DX9) applications targeted for release this year. The key issue with this game scene is that it is barely DX9."

NVIDIA:
"So, where do you find a true gamers’ benchmark? How about running actual games? Most popular games include a benchmark mode for just this purpose. Doom3, Unreal Tournament 2003, and Serious Sam Second Encounter are all far better indicators of current and upcoming game performance."

Also almost all of their statements are false.

1.Pixel shader 1.3vs1.4. They mention the only two games that use 1.3 and state that there are many more without citiing them. PS1.3 is NOT being used in Doom 3 ps1.4 IS. And they never moaned when ps 1.1 was used in the nature demo.

" PS 1.2/1.3 dont affect performance AT ALL. You dont get any performance gain over PS 1.1 which is why is does not make any sense to use them. and nearly no one does. Where as PS 1.4 is a subset of the functionality of PS 2.0 and basically Cuts the numer of passes needed to render compaitable scenes by 1/2. Last years nature demo even in the Nature test only used PS 1.1 even though it came out after the GF4. meaning they never cried at all that their precious PS 1.2/1.3 (that do nothing noteworthy) were not used. Even though the radeon 8500 got completely blown off and not properly supported in any of teh SE benchmarks." Hellbinder quote from rage3d.

2.Plus they are dissing flight simulation which I and many others find one of the best game genres of all time explified by a great game IL-2sturmovik. And WE are asking for more high level interception and bombing runs, which nvidia thinks inane.


The statement quoted above reaks of unprofessionalism. That fact alone regardless of the malicious intention of slandering the hard work of futuremark has completely turned me off of nvidia's products. I will never buy another one again.

You don't see ANY other companys doing this. ATI, AMD, INTEL, VIA, SIS, MATROX. NONE!

Nvidia is just pathetic.

Even my little nephew is more grown up.


Rogo

[Zeno]

Originally posted by Solomon
Isn't this sort of ironic. Considering ATi and their Quake3.exe debacle in the drivers. How everyone harped on them for this cheating bit. Now it seems that people are coming out of the wood work saying that Nvidia did indeed optimize drivers for the synthetic benchmark when before all of this no one really wanted to say such words. Now it seems people can't get enough of saying that Nvidia is mad because their drivers aren't optimized or what not for a specific benchmark.

It's weird. First Nvidia users wouldn't admit that Nvidia was indeed doing that. Now that this has happened you can't get enough of them saying that they are mad because their optimizations aren't happening right or what not.

It's like the Twilight Zone! :eek:

Regards,
D. Solomon Jr.
*********.com

You're misunderstanding what people's problem was with the Quack3 issue.

One way that graphics card vendors optimize their drivers for games is to watch the OpenGL or Direct3D calls that the game makes, and optimize those calls or combinations of calls within the driver. This is a legitimate thing to do. Any optimizations done here will benefit ALL programs that make those API calls.

However, what ATI did was watch for a specific game instead of specific API function calls, and then they actually lowered the image quality some to gain additional speed.

This is cheating in 2 ways: 1) Those optimizations won't apply to ANY OTHER GAME. If you're not playing Quake3, then the time they spent doing that work was wasted on you. 2) They're not really optimizations....they're image quality reduction. It's a totally different thing.

-- Zeno

[Solomon]

WoW! That's a pretty bold move to base a purchasing decision just because of a synthetic benchmark. If Nvidia whined, but their graphic cards were excelling in games you still wouldn't buy their hardware? Interesting philosphy.

Regards,
D. Solomon Jr.
*********.com

[Solomon]

Zeno,

Some call it cheating some call it a conflict with the hardware. I don't want to get into the worms of things again with this situation. I wanted to bring it up because it just seemed that's how people are responding to Nvidia's comments about not supporting 3DMark03. That people were like, "Well their drivers were optmized for the previous ones, just because they aren't now they are crying?" I've seen that in a lot of forums.

That's why i brought it up. I thought it was kinda funny to see people blatantly say that the drivers are optimized for the benchmark when before no one wanted to admit it or say it. Odd. Hehe.

Regards,
D. Solomon Jr.
*********.com

[jbirney]

For me I would be perfectly happy with 3dmark if it just drops the total score. I dont like how they wieght the score. I understand why, I just dont buy it.

I would rather see just the scores from each teast. And no final tally. Thats what I think makes 3dmarks neat and provided the most usefull info...

[jnd3]

Originally posted by Typedef Enum
I don't know who these guys are fooling, but it's certainly not me. nVidia rised to the top by doing that EXACT same thing...what the heck are they talking about? They have been tweaking their own drivers to sport significantly high Quake/3DMark numbers for years, and then using that data to demonstrate just how much performance their engineers could eek out of their drivers...
Wait, isn't Quake3 a game...? And aren't quite a lot of games are based on the Quake3 engine...?

:angel:

Cheers,
JND

[FrgMstr]

Originally posted by Solomon
Isn't this sort of ironic. Considering ATi and their Quake3.exe debacle in the drivers. How everyone harped on them for this cheating bit. Now it seems that people are coming out of the wood work saying that Nvidia did indeed optimize drivers for the synthetic benchmark when before all of this no one really wanted to say such words. Now it seems people can't get enough of saying that Nvidia is mad because their drivers aren't optimized or what not for a specific benchmark.

It's weird. First Nvidia users wouldn't admit that Nvidia was indeed doing that. Now that this has happened you can't get enough of them saying that they are mad because their optimizations aren't happening right or what not.

It's like the Twilight Zone! :eek:

Regards,
D. Solomon Jr.
*********.com


I think these statements show a general ignorance of the entire situation surrounding the issues above. Your comparison is off base.

What ATI did during the "Quack" issue and what you are seeing NVIDIA currently do, (and I think we are the only ones to prove this currently) are two totally different issues. I would think that someone that runs a site that caters to the 3d crowd would easily see the differences but obviously not.

ATI cheated on the QuakeIII benchmarks by putting QuakeIII specific optimizations in the driver. They did NOT optimize for the QIII engine (which is used by countless games), they only optimized for the game that is widely used as a benchmark, hence they cheated on the benchmark and ignored all the other games out there that could have "benefited" from the technology. That opens up a whole other argument that what they did was of no benefit at all, but rather just and exercise in turning down the overall quality to get better FPS.

NVIDIA optimized it drivers for the techniques that were specifically being used inside 3DMark03. Any game that uses those techniques will benefit from their changes.

So in theory, NV's changes could have far reaching effects that benefit the community at large where ATI's did not.

NV's argument here is that the changes they made in their driver benefit no one, as the techniques being used inside 3Dmark03 are not indicative of any game that we will see now or likely see in the future. They "won" the benchmark with their GFFX and they still don't like the benchmark.

It's only like the Twilight Zone if you don't understand what you are looking at.

(EDIT: DITTO Zeno)

[Hellbinder]

You're misunderstanding what people's problem was with the Quack3 issue.

One way that graphics card vendors optimize their drivers for games is to watch the OpenGL or Direct3D calls that the game makes, and optimize those calls or combinations of calls within the driver. This is a legitimate thing to do. Any optimizations done here will benefit ALL programs that make those API calls.

However, what ATI did was watch for a specific game instead of specific API function calls, and then they actually lowered the image quality some to gain additional speed.


This statement is really Nieve.

All video card drivers detect certain apps or function calls used by specific apps. Just like the recent Nvidia splash screen 3dmark score *debacle*

I am SICK of this bull**** spin you guys put on that quake issue. Ati came out publically and stated that Yes there are game specific optimizations, and yes it was a bug. The bottom line is it only affected 5 total textures in the game, affected the score by 5 FPS and lasted a total of 2 weeks before it was fixed.

Need i bring up the Nvidia 40.41 dets? or any of the other Driver false starts that were conveniently released only to benchmark sites just prior to other companies product releases, but never released to the public? how they only affected a few programs but adversly affected the majority of other sofware???

And you guys CONSTANTLY blow all the unethical **** Nvidia does off, and shout the *quak* issue from the rooftops. Well tht stops here and now. Its bull****.

[batterbrain101]

Game play is what matters when you get right down to it, most of us have known for awhile now that video drivers are optimized not only for games, usually they make damn sure that the drivers are optimized for peak performance with most popular benchmark programs. I've noticed none to no change in gameplay from the last 9 or 10 Det's I've installed, 3dmark usually increased though. Got to have the latest and "greatest" though, and hell running the benches and trying new drivers is fun to me. Now the most popular bechmark prog is all new and makes your system look really slow in points value (remember this happening when 3dMark2001 replaced the 2oo version?) so you can get that "my system is inferior" complex and run for new upgrades (which is also fun) and get the cash flowing. That's the way its always been and as long as techology advances, I know I'll be chasing it. Most of us NvNews members are members for that reason. anyway, thank you Muke for the best video card site (and tech news) around IMO. Nv news rules! http://smilies.sofrayt.com/%5E/w/super.gif

[FrgMstr]

Originally posted by Hellbinder
This statement is really Nieve.

All video card drivers detect certain apps or function calls used by specific apps. Just like the recent Nvidia splash screen 3dmark score *debacle*

I am SICK of this bull**** spin you guys put on that quake issue. Ati came out publically and stated that Yes there are game specific optimizations, and yes it was a bug. The bottom line is it only affected 5 total textures in the game, affected the score by 5 FPS and lasted a total of 2 weeks before it was fixed.

Need i bring up the Nvidia 40.41 dets? or any of the other Driver false starts that were conveniently released only to benchmark sites just prior to other companies product releases, but never released to the public? how they only affected a few programs but adversly affected the majority of other sofware???

And you guys CONSTANTLY blow all the unethical **** Nvidia does off, and shout the *quak* issue from the rooftops. Well tht stops here and now. Its bull****.

Wow, you got excited. All that ATI koolaid is going to your head bro, calm down.

ATI cheated and they got busted, you can SPIN that however you like.

I agree with you onthe 40.41 dets, that was bad business that we did not agree with. Also you did not see use those on [H]ardOCP.

To say what NVIDIA does business-wise gets blown off it actaully quite funny as I think they are under a big a microscope as anyone else. You seem to be too emotionally attached to the issues at hand and maybe stepping back for a moment and looking at the big picture will give you some clarity.

[FrgMstr]

Here are our thoughts on what needs to be done overall (http://www.hardocp.com/article.html?art=NDMw) in order to "fix" things. If we had solid tools to begin with, we would not be having this discussion. We of course would be arguing about something else totally! :D

[Hellbinder]

NVIDIA optimized it drivers for the techniques that were specifically being used inside 3DMark03. Any game that uses those techniques will benefit from their changes.

So in theory, NV's changes could have far reaching effects that benefit the community at large where ATI's did not.

NV's argument here is that the changes they made in their driver benefit no one, as the techniques being used inside 3Dmark03 are not indicative of any game that we will see now or likely see in the future. They "won" the benchmark with their GFFX and they still don't like the benchmark.

It's only like the Twilight Zone if you don't understand what you are looking at.


Bull**** Kyle.

Again for the same reasons I stated above. I am SICK of you guys playing favroties all the time. Enough is enough. You do everything with one eye open. Even if the Quak thing was a cheat, it is only one case out of the DOZENS of times Nvidia has pulled driver bull**** with fake launches, tweaks that only affect benchgmark apps etc etc etc. Besides which there is no doubt that they are outright CHEATING in some game benchmarks. or dont you fintd it odd that the GF4 has a higher score in SS:SE with 4x FSAA and 8x AF than the 9700pro. Or look at Quake benchmarks. The ones that use the 1.17demo to test score considerably higher than what is seen with the full retail release.

As For 3dmark03
They use vertex shader 1.1 routines 36 times. are you actually going to say that Simple Vertex shader 1.1 routines are NOT going to be used in future games? We are not in the future yet Kyle now are we. What you have allowed yourself to do AGAIN is be bought lock stock and barrel by Nvidia.

What you have done is embrase NVIDIAS view of the future of games and how THEY want games to be programed to support THEIR hardware better than everyone elses. You better step back and think a little deeper about what you are embrasing, and the road you are going down with this.

Are you really prepared to say that Nvidia has a lock on how future games will be coded? Further they are complete hypocrites. The Entire direction game design and coding is going is TOWARD single-textured Shader Rich designs. This has been getting pushed by ATi, Nvidia and Dx9. It is common knowledge. 3dmark is made to show ralative game performance in FUTURE games. Thus lots of single textureing. Lots of DX8.1 Vertex shaders, lots of PS 1.1 and 1.4.. and some DX9 code starting to appear. How can any HONEST person argue agaisnt that?

Futuremark is not supposed to show performance on CURRENT games. not at all. That is so self apparant its not even funny.

I am sick of the utter hypocracy. Will people ever stop allowign themselves to be brainwashed by Nvidias Cleverly worded PR spins?

Its not looking like it.