Alexander Jorias, Managing Director at Massive Development, furnished us with the follow FAQ in regards to the upcoming AquaMark 3
benchmark. They are based on questions that were sent in by those reviewing/testing the benchmark.
Q: Why is AquaMark3 more balanced than HalfLife2?
A: we don't want to comment on other developers or their applications. we are sure that valve has good reasons for what they are doing and why they are doing it this way.
Q: Did Valve do something wrong or is it too hard to tell?
A: you have to ask them not us.
Q: Do you think that there will be games out there that are crap on a specifc vendors hardware?
A: no. as a game developer you have to create revenues by selling your product to as much customers as possible. so it makes no sense for you to create a technology which gives a substantial amount of customers a bad experience. you may decide to implement special features for an oem version of your game. this makes perfect sense but is not an option for a benchmark. if you do not depend on the retail or any oem business and you are able to recoup your development costs from other sources you might be more flexible.
Q: Had any hardware vendor influence with the development of AM3?
A: as a developer of high end 3d engines and games we traditonally (have to) work close together with all major cpu/gpu vendors since years. they tell us about their current and coming hardware so that we can decide early enough where we will go technically from our side we provide them with valuable input. often their driver and hardware teams really get a harsh 'reality check' when we tell them how games do look from inside.
but we do not favour a specifc vendor simply because this could hurt the sales of our products because we might exclude customers. so we have to be solomonic when we design engines and technology. not because we are keen on being solomonic but we do want to give a good experience to all of our customers.
this is basically the same with all game developers. however, for a game you might decide to do a special version for oem which support a specifc hardware feature but this is not an option for a benchmark (and traditionally not for the retail version). as a game developer you have to implement your algorithms in a way that all customers do get an optimum gaming experience and the most out of their hardware. but because gfx cards do change fast it in general makes no sense to implement a specific algorithm especially for a specifc hardware generation. you have to find a general and efficient way.
Q: Why is driver X faster than driver Y?
A: driver and hardware are closely tied together and we think that any driver improvements which give an overall speed increase to the customer are good (no matter which vendor we are talking about). you have to remember, that we as developers use an api (dx9) which is generic and ideally solomonic. so we can not code to the metal and get the best out of a specific hardware. so the hardware vendor has the task to map the api optimally to the hardware. this may turn out to be a laboursome and painful task as you are aiming at a moving target (i.e. constantly chaning game engines). so you should think of the driver team as a constant service you as a customer get after you bought your hardware.
the development of a driver is a continous process. you have to notice that the source code size of a driver easily exceeds the source code size of a modern game! for this reason, there is always room for improvement. in many cases the improvement is necessary for reasons which are often underestimated. the problems for the drivers performance arise from hanges in the paradigm of game techniques which are on the other hand driven by the cpu performance. a change in game rendering techniques may dramatically change the driver demands. for instance contemplate the number of drawprimitive() calls which can be executed per frame. in previous games the average number of calls per frame was about 100-300. an unefficient implementation in the driver did not contribute significantly to its overall efficiency. with increasing cpu power and changing rendering paradigm (z first, multipass, materialdriven) the number of drawprimitive() calls per frame has multiplied tenfold. suddenly this new situation leads to a performance bottleneck in the driver which could could not have been foreseen. in this case the drivers code improvement becomes essential to get up-to-date. depending on the real situation this could easily lead to a performance increase which seems to be astonishing and somehow nonserious.
another unexpected performance increase simply comes from driver cleanup and refacturing. especially driver models which claim to support all generations of hardware within a single concept may suffer from compatibility restrictions. purifying the driver code from needless ballast may also lead to unepected performance increases.
Q: What is SVIST for and why it is so slow?
A: the svist technology is very smart, we have to guarantee that we color the pixels of a specific ps type WITHOUT rewriting or altering the original ps code. that's why it is very complex and slow. but we have to be very accurate and precise.
however svist is a key feature in terms of transparency for all users of am3. as you might read in our white papers we are convinced that the amount of rendered pixels of a specific ps type is more relevant to any game and especially a benchmark then just claiming that there are ps2.0 in the benchmark. there are benchmarks out there who i.e. implement trivial ps2.0 for particles which is insane. particles are typically rendered without any ps at all. with svist this is visualised for you. if you change the advanced settings you will notice that the svist visuals will change which reflect that the engine uses different code path for different situations.
besides we are able to tell you with svist the exact amount (absolute, relative and in percentage) of pixels drawn through a specific ps (or even simple texturestages like the particles).
we are convinced that svist is one of the coolest features of am3, however we are wondering if people will get the point about it because you have to think about it before you see it's beauty. we hope that there are enough smart guys out there that will tell the others about it.
Q: The AutomatedTestingTechnology is cool, but I do want to fully customize my runs!
A: no problem you can do this. when you set the advanced options your settings are written to a control textfile. by the command line options of the 'plus' version you can create a batch file an feed as much benchmark settinhgs to am3 as you want. this option is be described in your documentation. we implemented this especially for power-users.
Q: What about the excel sheet?
A: it is designed for visualisation of multiple measurements! Give it a try, you can use it oout of the box or take it as a starting point for you customized result visualisation.
Q: What is special about ARC?
A: ARC gives you more options and tools for comparison of your benchmark results than any other online benchmark database tool. just have a look at the many ways you can filter and set comparison options. it is the most powerful online database for benchmarks currently available.