I consider the recorded ambience part of the performance. Good luck finding recordings without it, whether Avery Fischer or Lexicon's finest. It's gotta be there.
I also agree that one reverberant field is company and two's a crowd. Since you can't get rid of the recorded ambience then controlling the listening room is the only option to reduce that mode of distortion. I continue to experiment with quasi-anechoic listening space with excellent results. It is natural to dismiss anechoic space for hifi listening, until you try it!
If you listen to studio recorded music, then the ambience is distortion free. There are no reflections in a reverb algorithm, unless added by the engineer. If you listen to recorded space, the room can vary from awesome to shit. Nothing you can do about it. But if you eliminate additional ambience in the listening room, your brain will adjust to the shit room and perceive the music recorded there as if you were there too.
Especially with stereo pair mics. Diffusion in the anechoic room would help you feel more comfortable when music is not playing, but would distort the music with (diffuse) reflections and reduce the illusion by holding your mind in the present time/space.
BTW, I recorded a string quartet in my listening room last year. I will try to make a short clip with a raw take versus the mastered version. It would be cool if I could play it back in the room and record what it sounds like compared to the original. Hmm, maybe I can do that, gotta find a quiet time after kids are in bed.