I feel that lyrics can only enhance the experience. Of course music is very subjective and different pieces of music do influence people in different ways.
I think that with certain genres like opera, one can be swayed by the music alone, but understanding the lyrics and plot or a certain character's frame of mind can be so much more rewarding but not knowing what the lyrics mean should not take listeners out of the moment. I remember Philip Glass was opposed to the Met Opera's practice of allowing the lights to be on during performances so people could read the libretti.
For some genres like music theater (Broadway), I feel the lyrics are indispensable because a lot of the impact depends on understanding the witty writing, such as lyrics by Sondheim, Lerner, or Tim Rice, for instance.
For pop or rock, lyrics might not mean as much. I remember misunderstanding lyrics to some rock songs for years. A few years ago I saw a Japanese technopop act - Perfume. I did not fully understand the lyrics to some of the songs, but enjoyed it immensely , nevertheless!