Originally Posted by xianthax
Except that he concludes that Distro X is slower than Distro Y based on the default desktop installs using the default file system settings based on performance differences in server tests.
If you revisit his tests and look at only the desktop benchmarks they paint an entirely different picture of the differences between distros.
If he wants to run server benchmarks he should use distros designed for server use with file systems tuned for the particular workload. Otherwise, he's not addressing real world usage patterns and his benchmarks are useless.
Any moderately capable server admin will use the proper file system, file system configuration and kernel configuration for the role of that server. Finding the ideal configuration for various server workloads isn't always easy but is something that Phoronix could be extremely helpful with, if they would setup a reasonable test.
Especially useless are his benchmarks across various kernel versions. He again uses the default kernel/file system configuration options and proceeds to conclude kernel x is slower than kernel y not due to any actual performance differences in the kernel, but rather changes to the default kernel configuration. The resulting information is not useful at all. If he wants to test kernel performance lock to 1 kernel / file system configuration and run the tests so we can actually see if the kernel code paths have improved.
In general he is exceedingly bad at setting up anything resembling a controlled experiment. His tests almost always have multiple uncontrolled variables making the resulting data nearly useless in making any worthwhile conclusion.
In this particular example he tested 1 single system with a mobile GPU, notorious for variations in implementation, and concluded that the entire driver was slower in general. Failed logic.
Phoronix is in a great position to provide some real value in linux performance testing. However, as long as they care more about sensational headlines than they do about setting up controlled tests with real value, they do more harm than good.
Agreed. Just look at the kernel test they did recently. They made a panic doomsday headline and within a week or so, the problem was fixed. This is 'testing' and welcome to the world were testing releases get broken and regress and testing it all is about catching these things, no making headlines about it.