I don't think it has anything to do with the way the tests are run. Audio appears to be a field where DBT as we know them simply don't work at all.
To run a double blind test properly can take hours to arrive at a conclusive single answer. We learned this back in the 80’s when the engineering staff at the recording studio I worked at decided to optimize the gear we had. That is what happens when there isn’t a ton of money to be able to buy “better” gear. The old POOGE process in application.
The impetus: we had an Otari 1/2” 30 ips mastering recorder that blew up the power supply. The four of us that had worked with it for years absolutely knew that machine inside out concerning sonics. A replacement power supply was over $1K back then, but worse yet getting one was at least a month out due to the shortage of two generation old technology. The option was to repair the power supply. When I tore the power supply down it was obvious that some of the components were minimal at best. I replaced the smoked transformer with the only one I could find quickly with one that had twice the current capability. The diodes that the TX took with it in its death throes were replaced with newer higher current versions. Smoked resistors were replaced with 10W wirewounds. Filter caps were doubled in capacity because I found them at Sandis National Labs salvage for pennies on the dollar - again it was what I could find. Rebuild took a week. When we did the cross our fingers test all of us were astounded at the improvement in SQ of that machine playing back our Reference Standard tapes. That is when we decided to begin upgrading the essential gear we had when possible.
We opted to use the jury method of auditioning parts and pieces to be used to upgrade the equipment. It didn’t take us long to determine that in order to achieve 90%+ repeatability in selection meant spending time doing extended listening to choose the best parts, from caps to opamps and transistors, etc.
Why?
Simple. Performance anxiety induced by making forced, hasty decisions. One of the techs designed and built a random selection switch that was a choice of three. It was the first ABX comparator any of us had seen. When a test subject has time to leisurely do multiple comparisons with the opportunity to take breaks to prevent ear fatigue, all of us were able to independently determine the best sounding parts with repeatability and unanimity in selection as to what was best.
Total upgrade took a year. From the original Otari victim through the outboard gear and finally the mixing desk...
The first Sara K GYPSY ALLEY album that got her a Chesky contract (another sordid story) was recorded on that gear. The MCI JH16 2” multitrack recordings were mastered on a shitty Sony U-matic 14-bit (14 bits on a good day) digital recorder. Even so, it still stands as a decent recording from the dawn of digital.
TMSAISTI...