As a solo tourer or commuter, I don't feel at all guilty if I hold up a few cars on narrow roads, as I don't have a choice but to be on those roads. Especially since I'm using the roads for their primary function, which is to go from point A to B. If traffic gets backed up more than several cars, I can pull off.
As a cyclist in a group, I'm using the road for pleasure and exercise. It's much different, and without the same moral high ground.