http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html “Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me..” -
J. Gorden Holt The concept of blind testing is valid, but only where the experiment can be controlled to eliminate variables that would skew the results. In audio, this would require that the system be transparent enough to discern important sonic attributes of the product under test, and also that the listening panel possessed adequate listening skill to discern the differences. These constraints make blind testing in audio a difficult proposition. In medicine the testing tool and the interpreter are one and the same, but the test can be ongoing with many samples and statistics draw the final conclusions which are usually still very nebulous. 5% difference between drug and placebo is often considered good enough to acquire FDA approval for the claimed benefit. 5% of a panel of 5 with no repetition is hardly statistically valid result - and yet, expecting 100% agreement on every blind test is unrealistic.
Also blind testing means quick testing. What sounds good for an hour, may be torture a month later after the brain adjusts to the change and begins assimilating the emotional and physiological effects of the sonic differences. So blind testing being short term is only useful for determining whether a difference can be heard, not for making judgement on the quality of differences heard.
As most ANers and other advanced audiophiles know, every change to an audio system makes an audible difference. Knowing that there is a difference without qualifying the difference is useless information. So there is minimal value to the individual audiophile in blind testing.
The other problem with any short term testing is that it occurs only on one system. Because of the reactive nature of electrical systems, and the high degree of sensitivity of the ear brain to discern differences as small as .00001% THD distortion, even quality judgements about a product are irrelevant unless the test system matches the consumers own system.
All of these excuses to invalidate blind testing are legitimate but could be overcome with effort and expense, as with any engineering problem. Pharmas figure a half billion and 5 years to get approval for a new drug. It is an enormous proposition to get statistically valid result from double blind testing even when variables facing audio testing are not present. Granted, an audio test is not a life or death proposition and there are many more variables in human metabolism than any machine. But professional audio reviewing interests hold fast to the excuses rather than seeking solutions to the technical challenges and employing blind testing because they fear losing control of their diocese by revealing the results of blind testing.
The best use of blind testing is as an excuse to gather with friends and drink beer and have some laughs when the curtain is pulled back.
