I agree with the recommendation to go ahead and ride with that faster group, you get better at any activity by being surrounded by those who are better than you.
I sandbag in the gym all the time, put me in a training sessions with guys who are stronger than me, and I start performing. It's easier to push yourself in a group setting, your ego will make you keep going.