I'll throw in my two cents here since I do have a performance oriented bike with downtube shifters - it's my 'office bike' and it gets used for group rides, both fast and casual. It's not my primary road bike, but it does get ridden hard. Downtube shifters are fine for riding in a group with others when they're on integrated if you have the handling skills to hold your line while shifting, you're a strong enough rider that being in less than optimal gearing on occasion doesn't hold you back, and you're mindful of how using them changes your shift pattern and timing. A more important thing to be concerned with when riding with people who are on different bikes are the differences in brakes and how that translates to stopping distance and brake timing.
Picture: