View Single Post
Old 05-28-10, 06:29 PM   #29
xianthax
Registered User
 
Join Date: Dec 2008
Posts: 38
Default Re: Performance regressions with 256.25 driver?

Quote:
Originally Posted by Licaon View Post
that's the point: to test file systems at the DEFAULT settings ( provided by the distro ) because a generic user will use the defaults while the specialised ( server, workstation, etc ) user will know and want to optimise the settings as s/he sees fit anyway
Except that he concludes that Distro X is slower than Distro Y based on the default desktop installs using the default file system settings based on performance differences in server tests.

If you revisit his tests and look at only the desktop benchmarks they paint an entirely different picture of the differences between distros.

If he wants to run server benchmarks he should use distros designed for server use with file systems tuned for the particular workload. Otherwise, he's not addressing real world usage patterns and his benchmarks are useless.

Any moderately capable server admin will use the proper file system, file system configuration and kernel configuration for the role of that server. Finding the ideal configuration for various server workloads isn't always easy but is something that Phoronix could be extremely helpful with, if they would setup a reasonable test.

Especially useless are his benchmarks across various kernel versions. He again uses the default kernel/file system configuration options and proceeds to conclude kernel x is slower than kernel y not due to any actual performance differences in the kernel, but rather changes to the default kernel configuration. The resulting information is not useful at all. If he wants to test kernel performance lock to 1 kernel / file system configuration and run the tests so we can actually see if the kernel code paths have improved.

In general he is exceedingly bad at setting up anything resembling a controlled experiment. His tests almost always have multiple uncontrolled variables making the resulting data nearly useless in making any worthwhile conclusion.

In this particular example he tested 1 single system with a mobile GPU, notorious for variations in implementation, and concluded that the entire driver was slower in general. Failed logic.

Phoronix is in a great position to provide some real value in linux performance testing. However, as long as they care more about sensational headlines than they do about setting up controlled tests with real value, they do more harm than good.
xianthax is offline   Reply With Quote