At 3-5 feet behind there's considerably less drafting happening than at 3-6 inches which would be more typical of actual drafting. Also a lot less danger of "we all fall down". Anyway, usually if contact occurs, the rear rider is the one going down. The danger to the front rider, especially at 3-5 feet, is over-hyped.
That said, I still always announce my presence if I'm going to grab someone's wheel, or hang behind them by less than a bike length or so. If they show signs of not wanting that to happen, it's time to pass (and hopefully they aren't going to be a dick about that).