About Aquamark 3.. (I appologize in advance)

[Hellbinder]

This was posted at the massive website.

Statement of Massive Development regarding AquaMark3 / IHV cooperation & cheating attempts

As a developer of interactive high-end 3D gaming applications and benchmarks we obviously have to collect in-depth Know-How about a specific hardware from it's manufacturer.

This type of cooperation is common practice for all major developers. It serves the sole purpose to utilise the latest technologies in the correct way and thus giving the customer the best gaming and performance experience possible.

You may decide to support a specific hardware platform or a specific hardware feature by a game application. However this not an option for any serious benchmarking application.

While developing AquaMark3 we take precautions in a way so that all IHV's (Independent Hardware Vendors) with a vital interest in AquaMark3 are treated equally. So there is no reason for those vendors to conduct any cheating attempts on AquaMark3.

Once a benchmark is released there are obviously multiple possibilities to cheat its results (both for any IHV and any user). We took precautions to prevent and detect cheating attempts in this post release phase and during the lifetime of AquaMark3. Unfortunately we can not prevent this in all cases by the application itself.

The most powerful anti cheating tool will be the AquaMark Result Comparator (ARC), the online database and the AquaMark3 forum. We will immediately inform all customers about our evaluation of the latest cheating attempts (if they should happen).

As a developer of high end 3D gaming applications we will of course be very specific and accurate when we have to judge if a specific optimization is of general interest or if it only targets AquaMark3 in a questionable way.

We are always open to discuss our policy with interested users in this forum.

Alexander Jorias / Managing Director Massive Development
Ingo Frick / Technical Director Massive Development

Here is the problem with their entire statement, and how it contradics what they have already said.

They origionally stated that the benchmark would be *PURE DX9* and no optimizations for vendors allowed. Even rendering results as *invalid* if any code changes were detected.

Of course what they are not saying and what is obvious at this point. If you think about it. Any *optimizations* that give an unfair advantage to one IHV at the detrement of others Included in the code prior to release... well.. will *never* be seen as a cheat. Or even known about unless someone goes public with it.

Which raises an interesting Question. *Real* Dx9 shaders in HL2 according to valve run 5-10x faster on R350 than Nv35. in fact currently *ALL* benchmarks showing PS2.0 performance runs *considerably* slower on current Nvidia hardware. Thus it stands to reasosn virtually any basic DX9 shader is going to be considerably faster on ATi hardware. How is that suddenly going to change for Aquamark3??

Which leads me to two possabilities, Nvidia has paid money, or used similar tactics that they did to Futuremark only ahead of release. To pressure Massive into this new path. Or, looking at the history between the two companies. They were in cahoots with Nvidia from day one and this is just a ploy to draw attention away form it by making it sound legit.

Its desturbing to me. Especially considering all the earlier statements about *PURE DX9* and no IHV optimizations made earlier by this same company. What drew my attention to this was this post and Thread at Rage3d.

I just talked to a friend of mine who had the chance to meet one of the Massive guys, and he said he had a very interesting conversation.

Among other things the Massive guy told him that Nvidia had already optimised/hacked for the original aquamark/Aquanox game.

But then comes another thing, namely Nvidia is trying to make Massive include commands and routines in Aquamark 3 which will run slower on ATI cards, because they need an additional cycle, but will run in one cycle on GFFXes.

My friend said he didn't know whether the Massive guys are actually doing this, but it could destroy the impartiality of the benchmark right from the start.

So maybe someone should look at this once the benchmark is out and the results look fishy.

From wich some forum members went to massives site and *politely* asked what the deal is with optimisations like this. From which they were banned fromt eh site and the threads Deleted. shortly after that the above statement was posted by massive.

Wich should tell you that its in all likelyhood True. Massive is putting in special code that favors Nvidia before the benchmark is even released. Which imo invalidates it as a *fair and unbiased test*.

Mind you the main problem I have is not with Nvidia Jockeying for a little special support. I expect that. What ticks me off is the statements origionally made by massive about *Pure DX9 Code* and no optimizations for vendors. They made big PR statements about this back in the heat of the 3dmark03 fiasco. Yet what it looks like now, at least to me, is nothing but an intentional benchmark written to favor Nvidia.

I know I am going to get flamed hard by some people here for this. But i simply wanted to raise awareness and make people think in general terms about what really is fair, and how completely Screwed up the benchmarking *business* is.

[The Baron]

I know I am going to get flamed hard by some people here for this. But i simply wanted to raise awareness and make people think in general terms about what really is fair, and how completely Screwed up the benchmarking *business* is.
Amen. Don't believe what you hear until somebody you know and trust tells you it's true.

[Hellbender]

/Flame On
Sorry but that's bull****. Read their technical documents and you will notice that they claim they have a real world game engine driving this benchmark. They can't be dx9 only because they implented fallbacks to dx7 and dx8 in Aquamark3 and that's dramtically different to 3DMark03. As long as Aquamark3 is not released we will have to wait. However from what I can see we might have two very interesting benchmarks each with a different focus at our disposal and that's a indeed a nice thing to have :D

Maybe this heals your fears:

While developing AquaMark3 we take precautions in a way so that all IHV's (Independent Hardware Vendors) with a vital interest in AquaMark3 are treated equally. So there is no reason for those vendors to conduct any cheating attempts on AquaMark3.

From what I know 'Equally' describes 'Equality'!?
/ Flame Off

You have an interesting nick ;)

[digitalwanderer]

Originally posted by Hellbinder
From wich some forum members went to massives site and *politely* asked what the deal is with optimisations like this. From which they were banned fromt eh site and the threads Deleted. shortly after that the above statement was posted by massive.
ALWAYS a bad sign... :(

[Matthyahuw]

this is why I think it's going to be nearly impossible to bench ATI and nV cards side by side any longer...

They each have their own way of doing things, and no matter how fair you make things, it's not gonna be fair enough for the fanboys on either side...

[druga runda]

why impossible, why not use FRAPS... and that's about it... and ignore benchmark programs, at least people will know which game runs better than the other, + post screenshots of the game so people can see how it looks, and here you go.,.. even better if big websited have bandwith available screen a 2-3 minute FRAPS demo that's all... (even though taking it a little too far bandwith wise)

SO that's it, people will find out what real games play like, and what the image sacrifises are...

Benchmark only programs were a bad idea from the beginning, even though they sound attractive to users as they can compare easily, and to companies as they can distort reality in a easier and more controlled manner.

[Ruined]

Originally posted by Hellbinder
Which leads me to two possabilities, Nvidia has paid money, or used similar tactics that they did to Futuremark only ahead of release. To pressure Massive into this new path. Or, looking at the history between the two companies. They were in cahoots with Nvidia from day one and this is just a ploy to draw attention away form it by making it sound legit.

Or, it's just the best way to handle programming for two completely different pieces of hardware. Doom3 will use vendor specific optimizations, as will Half Life 2. It is becoming standard practice now, since the two major competitors have very different hardware. Conspiracy theories aside, game devs don't want to shut out half their audience, so it makes sense to optimize if the hardware being offered is totally different. And, if benchmark companies don't follow suit with similar optimizations, then the benchmarks won't be accurately predicting how the cards will perform in the most popular games, since they will be using these optimizations - and if a benchmark can't predict with accuracy how a 3D gaming card will perform in games, what is its purpose? In other words, so long as Nvidia and ATI remain to have very different hardware, it is likely you will see developers incorporate vendor specific optimizations in their games. In the end, game companies want money, and will program for a piece of hardware even if it is nonstandard or difficult to program for. See the Playstation2 as proof. :)

[-=DVS=-]

And all that started from Nvidia releasing crappy hardware with alot of hype , so now they trying do damage control and force companys to optimize for not standart DX9 hardware , if they would have made decent product like ATI did with MS DX9 spec we wouldn't have all these threads now :o

That is number one reason why PC have so many problems , alot of companys with they own spec :rolleyes: Microsoft tried to make everyone use same spec and make all hardware 100% compatible , but NO Nvidia had to come up with they own sh** makeing life harder for developers and us USERs

Well atleast CPU's working alright either its P4 or AthlonXp same code 32bit risc always same output same quality :p

[Ruined]

Originally posted by -=DVS=-
And all that started from Nvidia releasing crappy hardware with alot of hype , so now they trying do damage control and force companys to optimize

You are making it sound like it's illegal to refrain from optimizing a game for Nvidia cards. It is legal. But game companies are smart, want to get the most money, and therefore would want to get their game running nicely on both of the dominant video card manufacturers' products, so that gamers are pleased with the company's game. Not to mention if Nvidia or ATI throw a couple of extra bucks their way to implement optimizations, they get more money that way, too. Therefore it would be in a developers best financial interest to optimize for both Nvidia and ATI cards. In the end, though, it is their choice, and currently it looks like game companies are choosing to implement manufacturer specific optimizations.

P.S. -- The FX5900 is far from crappy, too - it's just not as standardized as ATI's offerings.

P.S. #2 - How exactly does Nvidia using less standard hardware make things harder for users? We don't do any of the coding or marketing... You could even argue that Nvidia's drivers which often prevent glitches from occuring in games when the hardware does not support it (i.e. AA in Splinter Cell) makes things easier for the user.

[digitalwanderer]

Originally posted by Ruined
You are making it sound like it's illegal to refrain from optimizing a game for Nvidia cards. It is legal
Yup, and I'm sure it'll catch on just as much as Glide did... :p

[Skuzzy]

There is a perfectly valid way to compare these video cards and make it apples-to-apples.

It's just that the current methodologies are lacking.

Sit tight..it's coming (never fast enough, but hang in there)

[StealthHawk]

Originally posted by Ruined
Not to mention if Nvidia or ATI throw a couple of extra bucks their way to implement optimizations, they get more money that way, too. Therefore it would be in a developers best financial interest to optimize for both Nvidia and ATI cards.

You're advocating IHV's paying off ISV's for support? Geez. What happens to people who own Trident, SiS, or Matrox cards then? They get no optimizations and the game runs/looks like crap because the companies either aren't willing or can't afford little bribes to every ISV?

If IHVs want to help developers of their own volition or at the developer's request I'm all for it. I don't believe that IHV's buying features is the right move. How long will it be until an IHV pays an ISV for exclusive content? That will be bad for the industry.

edit: Oh yeah, might I remind everyone of all the HL2 rumors? First we heard NVIDIA paid Valve to make it NVIDIA exclusive. Then when we heard about the FSAA issue some NVIDIots hypothesized that ATI bought FSAA support. Sure it was all bull this time around.

[Ruined]

Originally posted by StealthHawk
You're advocating IHV's paying off ISV's for support? Geez. What happens to people who own Trident, SiS, or Matrox cards then? They get no optimizations and the game runs/looks like crap because the companies either aren't willing or can't afford little bribes to every ISV?

The 'optimizations' aren't quite as radical as you make them out to be.


If IHVs want to help developers of their own volition or at the developer's request I'm all for it. I don't believe that IHV's buying features is the right move. How long will it be until an IHV pays an ISV for exclusive content? That will be bad for the industry.

The problem with that thinking is that assuming there is more than one major player (which there is), the amount a particular card manufacturer would have to pay to make up for the loss of sales a publisher would experience (along with the bad PR) for the other major player's card would be extrodinary, which is why a full exclusivity contract would likely not happen, so it's not something to really worry about at this point. What I think *would* be a good thing is if there were exclusive graphical features for cards if the cards can handle it - for instance, I've read that the FX 5800/5900 have much more flexible shaders than the Raedon cards (which in turn has lead to the lower performance when "unoptimized" as we have seen in many benchmarks). However, if some of the extra features of these shaders were actually used in a game instead of thrown out the window for performance (i.e. "optimized") to meet the current minimum standard that games use, I think that would be cool.

[StealthHawk]

Originally posted by Ruined
The 'optimizations' aren't quite as radical as you make them out to be.


Untrue. We have already heard from many developers how it is hard to get performance out of NV3x's shaders. We have heard claims that HL2's shaders run significantly slower on NVIDIA cards than on ATI cards. We have heard of the programming difficulties from the STALKER devs, and why they had to get assistance from NVIDIA. We have heard that on the ARB2 pathway Doom3 runs twice as slow on NV30 as on R300.

Think about it. If NV3x was made by Trident, how much do you think game developers would care about it's architectural intracacies. I'm betting not much. They'd let the hardware fend for itself. Even hardcore NVIDIots have acknowledged that NVIDIA will get custom treatment because of their marketshare. ie, that devs won't sit around and let performance be piss poor on NVIDIA cards because there are lots of NVIDIA owners. If it had been Parhelia that was a programming nightmare I doubt that most devs would give it a second thought.

An optimization that makes something run twice as fast(ie, NV30 path vs ARB2 path in Doom3 for a concrete example) is nothing short of radical.

[bkswaney]

Originally posted by StealthHawk
Untrue. We have already heard from many developers how it is hard to get performance out of NV3x's shaders. We have heard claims that HL2's shaders run significantly slower on NVIDIA cards than on ATI cards. We have heard of the programming difficulties from the STALKER devs, and why they had to get assistance from NVIDIA. We have heard that on the ARB2 pathway Doom3 runs twice as slow on NV30 as on R300.

Think about it. If NV3x was made by Trident, how much do you think game developers would care about it's architectural intracacies. I'm betting not much. They'd let the hardware fend for itself. Even hardcore NVIDIots have acknowledged that NVIDIA will get custom treatment because of their marketshare. ie, that devs won't sit around and let performance be piss poor on NVIDIA cards because there are lots of NVIDIA owners. If it had been Parhelia that was a programming nightmare I doubt that most devs would give it a second thought.

An optimization that makes something run twice as fast(ie, NV30 path vs ARB2 path in Doom3 for a concrete example) is nothing short of radical.


This is the good thing about owning Nvidia. Game makers have
to make there game run good on NV cards or they look like fools. ;)
Every person I know that games around here uses nvidia cards.
Every system I've built and sold had a Nvidia card in it. Why? The customer ask for it by name. The only prson I've sold a ATI card to is my brother in law. He has a AIW 9700Pro. He loves it. But he is a big video head. ;)

[ChrisRay]

Originally posted by StealthHawk
Untrue. We have already heard from many developers how it is hard to get performance out of NV3x's shaders. We have heard claims that HL2's shaders run significantly slower on NVIDIA cards than on ATI cards. We have heard of the programming difficulties from the STALKER devs, and why they had to get assistance from NVIDIA. We have heard that on the ARB2 pathway Doom3 runs twice as slow on NV30 as on R300.

Think about it. If NV3x was made by Trident, how much do you think game developers would care about it's architectural intracacies. I'm betting not much. They'd let the hardware fend for itself. Even hardcore NVIDIots have acknowledged that NVIDIA will get custom treatment because of their marketshare. ie, that devs won't sit around and let performance be piss poor on NVIDIA cards because there are lots of NVIDIA owners. If it had been Parhelia that was a programming nightmare I doubt that most devs would give it a second thought.

An optimization that makes something run twice as fast(ie, NV30 path vs ARB2 path in Doom3 for a concrete example) is nothing short of radical.

Are you sure you understood what he said? I'm not so sure. Optimising for the Nvidia cards is mainly about optimising properly for optimal use of its registers..



I really doubt either company is buying off game developers. I can imagine most game developers accept the hardware manufacturers help with open Arms.

[-=DVS=-]

I don't see how it is good if developers have to waste time and specificly code for NV30+ to perform good, when you program for DX9 game you don't need to make anything extra to perform good on R300+ line , cose its already useing optimal path and spec of DX9 , and now comes Nvidia where it performs poor and need special atention to perform half decently , explain to me how good this can be ? :o

Second point , if game devs diviate from DX spec or OGL for that matter and start useing ATI's and Nvidias specific paths , no new competitor can enter 3d card race or all games made useing exclusive paths will run like S*** :rolleyes: :mad:

Glide was ok if you had Voodoo , what if you didn't had it , games looked bad :( , Offcourse if there would be only one company with one Glide like software it would be no problem except no competition equals monopoly big prises :p

They should make cards work and use one code path but be different in speed and prise like CPU's , remainning competative and offering alot of options ;)

[StealthHawk]

Originally posted by ChrisRay
Are you sure you understood what he said? I'm not so sure. Optimising for the Nvidia cards is mainly about optimising properly for optimal use of its registers..

Understood what who said?

re NVIDIA shader optimization, it is obviously not that easy(especially when past NVIDIA drivers do not have full functionality exposed, and when they are rendering things incorrectly) because the STALKER guys had to seek NVIDIA help. Also I am not sure how easy it is to optimize for register usage...

Take a look at how many driver revisions and how many months passed before NVIDIA was finally able to optimize 3dmark03 with no loss of IQ. Even now we don't know exactly what the shaders are running in 3dmark03...is it full FP32? A mix of FP32/FP16? A mix of FP32/FP16/FX12? Only NVIDIA knows, and I don't think they are gonna ever tell us.

I really doubt either company is buying off game developers. I can imagine most game developers accept the hardware manufacturers help with open Arms.

I agree, I don't think anyone is buying of game developers either. I was specifically responding to a comment made by Ruined:Not to mention if Nvidia or ATI throw a couple of extra bucks their [ISV's] way to implement optimizations
To which I said that I don't think it is a good thing to have IHVs buying off ISVs. Because it is impossible to know where it will end. To support my POV I provided two actual conspiracy theories which didn't pan out.

[ChrisRay]

Originally posted by StealthHawk
Understood what who said?

re NVIDIA shader optimization, it is obviously not that easy(especially when past NVIDIA drivers do not have full functionality exposed, and when they are rendering things incorrectly) because the STALKER guys had to seek NVIDIA help. Also I am not sure how easy it is to optimize for register usage...

Take a look at how many driver revisions and how many months passed before NVIDIA was finally able to optimize 3dmark03 with no loss of IQ. Even now we don't know exactly what the shaders are running in 3dmark03...is it full FP32? A mix of FP32/FP16? A mix of FP32/FP16/FX12? Only NVIDIA knows, and I don't think they are gonna ever tell us.



I agree, I don't think anyone is buying of game developers either. I was specifically responding to a comment made by Ruined:
To which I said that I don't think it is a good thing to have IHVs buying off ISVs. Because it is impossible to know where it will end. To support my POV I provided two actual conspiracy theories which didn't pan out.


I was referring to to what ruined said. It didn't make sense when he said radical. So I was wondering if that was what your reply was.

I dont feel optimal usage of Geforce FX's register values is "radical". Do you?


I think it's fair to say That Stalker actively seeked help from Both ATI and NVidia, I'm surely most ISV would rather make optimal use of both cards.

And I was actually responding to ruined about them buying off ISV. Seems extremely unlikely to me.

[StealthHawk]

Originally posted by ChrisRay
I was referring to to what ruined said. It didn't make sense when he said radical. So I was wondering if that was what your reply was.

I dont feel optimal usage of Geforce FX's register values is "radical". Do you?

Well from what I understand about the gfFX that is not your only worry. You also have to worry about type juggling; using partial precision and maybe even FX12. It might not warrant "radical" but I also don't think it is a small problem. e.g. Shadermark even totally using pp is just as slow as pure FP32.

I think it's fair to say That Stalker actively seeked help from Both ATI and NVidia, I'm surely most ISV would rather make optimal use of both cards.

Yes that's true, but I remember them saying they specifically asked for help from NVIDIA because it was hard to code for.

edit: Here we are:

Discussion (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=9440)

Hi,

In fact, the graphics engine is developed on Radeon9700
We demand special support from FX driver guys, 'cause in pure/standart
DX9 it's impossible so run the engine on FX, at the moment.
Yes, FX will be slightly faster, but in the margin of several
percentages...

So, don't worry, your card will be excelent performer in
S.T.A.L.K.E.R.

--
Best regards,
Oles V. Shishkovtsov
GSC-Game World
oles@gsc-game.kiev.ua

I think that comment backs up what I was saying.

edit: So no, I don't think that just the optimal register usage is "radical." But combined with other problems I mentioned(incorrect rendering due to NVIDIA optimizations or impartial functionality and not fully exposed drivers) it probably can be.

[ChrisRay]

Originally posted by StealthHawk
Well from what I understand about the gfFX that is not your only worry. You also have to worry about type juggling; using partial precision and maybe even FX12. It might not warrant "radical" but I also don't think it is a small problem. e.g. Shadermark even totally using pp is just as slow as pure FP32.



Yes that's true, but I remember them saying they specifically asked for help from NVIDIA because it was hard to code for.

edit: Here we are:

Discussion (http://www.nvnews.net/vbulletin/showthread.php?s=&threadid=9440)



I think that comment backs up what I was saying.

edit: So no, I don't think that just the optimal register usage is "radical." But combined with other problems I mentioned(incorrect rendering due to NVIDIA optimizations or impartial functionality and not fully exposed drivers) it probably can be.

Ahh yes Well FX 12, FP 16 ect were part of the registers I was talking about. So thats understandable.

Interesting letter you posted. I have never seen it before. yet Strangely I remember that thread :)

[Ruined]

It makes sense that the Nvidia shaders may require additional programming knowledge, simply because they are more complex and flexible (and hardware wise near-VS/PS3.0 featurewise, as opposed to ATI's shaders which are on the level of VS/PS2.0 - though I don't think this has been fully exploited in the Nvidia drivers yet) than the competitors that devs have been using up to this point. Again, though, this is hardly a "radical" optimization that will destroy performance for other brand video cards - it will just make the game run/look its best on an Nvidia card as opposed to if it was unoptimized...

[digitalwanderer]

Originally posted by Ruined
It makes sense that the Nvidia shaders may require additional programming knowledge, simply because they are more complex and flexible
You forgot to mention, "not to mention a good deal slower." to this sentance. :)

[-=DVS=-]

Originally posted by Ruined
It makes sense that the Nvidia shaders may require additional programming knowledge, simply because they are more complex and flexible (and hardware wise near-VS/PS3.0 featurewise, as opposed to ATI's shaders which are on the level of VS/PS2.0 - though I don't think this has been fully exploited in the Nvidia drivers yet) than the competitors that devs have been using up to this point. Again, though, this is hardly a "radical" optimization that will destroy performance for other brand video cards - it will just make the game run/look its best on an Nvidia card as opposed to if it was unoptimized...


Explain more flexible ? you mean they can do more instructions then Radeon9700 Pro ? that is true however Radeon9800+ have unlimited shader instructions GFFX have its limits at around 1000+ or so and OpenGl 2 requires unlimited shader instructions its part of the feature , realy hard to se how GFFX is more advanced :rolleyes: sounds like simple hardware but to complex to use :p

[Ruined]

Some charts from Beyond3D here, a bit old, but since there hasnt been massive changes from r300->r350 or nv30->nv35, still somewhat valid. Best I can do at least, in terms of finding A/B feature comparisons between the two cards, though I know both manufs have added features in their new cards (probably biggest change is 256bit memory in nv35).

If anyone has more recent feature comparisons that would be cool.

As for speed, from what I've read on some of the programming boards is that the NV3x chipset is a lot more flexible and complex than the R3xx chipset. However, this complexity doesn't pay off in today's games (and in fact is slower run without optimization because of it), but will actually be much faster and be able to do more things in future games/apps. How long it will take to get to that point is unknown.

ATI has delivered a product similar to 3dfx when they were going up Nvidia back in the Voodoo2 days. 3dfx's product was less advanced than Nvidia's and didnt look as nice, and because of that it ran the games of the time faster. Seems to be similar to what is happening here with ATI and Nvidia. From the looks of PS/VS3.0, ATI will have to go the same route Nvidia is going now, just like 3dfx had to switch to 32bit color and 2048x2048 textures, stencil buffers, etc eventually. The question is, will Nvidia convince devs to use some of that shader complexity before NV35 and R350 are irrelevant?

http://www.reflectonreality.com/images/b3d1.gif
http://www.reflectonreality.com/images/b3d2.gif
http://www.reflectonreality.com/images/b3d3.gif
http://www.reflectonreality.com/images/b3d4.gif
http://www.reflectonreality.com/images/b3d5.gif