Join Date: Jul 2002
Location: Calgary, Ab
more from www.extremetech.com
The 2003 version of FutureMark Corp.'s 3DMark graphics benchmark was criticized by executives at Nvidia as unrealistic, who added that they had decided not to assist the developers in their beta testing.
FutureMark's "benchmark for gamers," 3DMark, is one of the leading "synthetic" benchmarks to test 3D applications. Testing agencies, such as ExtremeTech, routinely use a mix of application-based and synthetic benchmarks to test a product's real-world and theoretical performance under different constraints.
Nvidia executives said they had tolerated previous 3DMark versions, but that FutureMark's latest effort didn't give an adequate indication of real-world performance. However, FutureMark executives said they had moved their benchmarking suite in the right direction.
"High-end 3D effects are no longer the sole domain of serious gamers, but are becoming ubiquitous in mainstream applications," said Tero Sarkkinen, executive vice president of sales and marketing for FutureMark, in a statement.
"Since the release of the first version of 3DMark in 1998, we've been continually striving to improve our benchmarks to help users maximize their experience and PC performance," Sarkkinen added. "3DMark03 meets this goal by supporting DirectX 9.0 and providing detailed 3D performance measurements for existing systems, while offering challenging, new tests for current, cutting-edge hardware and technologies yet to be released."
In previous versions of 3DMark, like the current version, FutureMark crafted a custom 3D engine to run the tests; part of the 3DMark 2001 code was used to create the game Max Payne. So far, no developer has publicly announced its intention to use the engine in a commercial game, and a white paper developed by FutureMark says that the developer created a lightweight DirectX 9.0 wrapper for the new code base.
That, Nvidia executives said, meant that any assistance that the company would have given FutureMark would have been wasted.
"The reason that we're not all gung ho about it is that (3DMark'03) is not representative of (actual) games, nor is it a good benchmark," said Tony Tamasi, senior director of desktop product management at Nvidia. "That means Nvidia has to expend effort to make sure it runs well on our hardware. All that energy that we spend doesn't benefit the user. None. Zero. All that effort doesn't go to benefit any game, either. That's kind of depressing."
For that reason, Nvidia didn't participate in the beta testing of the 3DMark benchmark, only doing some last-minute driver tweaks, Tamasi said.
"Being a beta partner at all required us to pay money to Mad Onion [the previous name for FutureMark] and that seemed really wrong," Tamasi said. "Don't get me wrong -- that's a tough business. I totally appreciate the quandary that MadOnion is inů But from our perspective it didn't make sense for Nvidia to give them money for something we didn't believe in."
Specifically, Tamasi said he objected that Futuremark apparently chose to emphasize single-textured pixels in the benchmark's four tests, while previous versions had pushed multitexturing. Tamasi also criticized FutureMark's use of older version pixel and vertex shaders, and the benchmark's heavy emphasis on running and rerunning vertex shader operations -- 36 times, by his count.
Another hardware testing website, HardOCP.com, said Tuesday that it would not to use the technology in future tests of graphics cards. However, FutureMark claimed that testing shaders was critical to assessing a product's capabilities.
"Pixel shader programs, unlike their vertex counterparts, require hardware support," the company's white paper states. "Vertex and pixel shaders have become an important part of 3D graphics and are accordingly featured prominently in 3DMark03. One of the game tests, the DirectX 9 showcase, uses 2.0 vertex shaders and 2.0 pixel shaders. All other games tests use 1.1 vertex shaders. The DirectX 8 game tests use 1.4 pixel shaders if available; otherwise they default to 1.1 pixel shaders."
FutureMark allows free downloads of its standard benchmark, but charges $39.95 for individual copies or $250 for a corporate license to a "professional" version.
The lack of support from Nvidia is telling. Jon Peddie Research estimates that approximately 53 million PC graphics devices shipped from nine suppliers in the fourth quarter of 2002, a 13 percent increase over the previous quarter, and a market that was, somewhat surprisingly, controlled by Nvidia. Nvidia led the market with a 32 percent market share, followed closely by integrated-graphics powerhouse Intel, with 28 percent. ATI finished third with a 19 percent share, followed by Via and SIS. Matrox, Trident, Silicon Motion and 3Dlabs each controlled 1 percent of the markt, or less.
So what is Nvidia's choice for benchmarking? "Games," Tamasi said. "Use games."
In the fight between you and the world, back the world.