THe way I see it is rather simple. For I am a simple person. If a person is able to latch onto the last wheel of a fast paceline (and more importantly- stay there until he finds himself upfront) he would most likely be familiar enough with pack riding to not be too put off by being yelled at due to some error. In that sense I agree with the OP completely.
You are training and are riding a race pace for correct training. Do yourselves a huge favor and create a club and design and issue jerseys. It's been mentioned before i know. But it would thwart at least 70% of all your woes.
I KNOW that if I saw your mini peloton barreling down the street with all riders wearing the same jersey- I would NOT attempt to jump in (no matter how much fun I know I'd have).