Lots of things have been said. It's also a bit about individual expectations, especially for the stronger riders. If the stronger riders want real exercise, they should consider splitting that component off from the ride.
When I go with slower friends, I consider it a social ride. No-drop. The goal is to keep everyone happy and that nobody feels like they are the hold-up for everyone else. If I cause someone to feel like they are the hold-up, I have failed at being a good group member and friend. To get good exercise for myself, I join other groups in addition to these rides or go solo.