I've encountered the same in fast group rides. I do a fair amount of pulling up front, some people are content to hang out in the back (or aren't strong/confident enough to take pulls). I don't mind pulling, but I don't want to slide all the way to the back either. If a spot isn't opening up towards the front, I'll just point to the right and ask the person if I can slide in. Most people are more than happy to let someone else stay up front and keep pulling, so they let me in. As mentioned above, I would imagine a race scenario would be different, but I haven't done any group racing yet, just TTs.
I just look at it as getting a better workout, the people hanging out in the back aren't going to get any stronger.
