Do you have the same issues with the rest of your riding stuff? Tires? Shoes? Pedals? Jerseys? Bibs/shorts? Seat Bag? Water Bottle? Saddle? Cell phone in your pocket? Ipod?
No dilemma for me. Its a modern convenience, and one that does no harm to the bike (no grinding, no drilling, no modifying at all).