I won't let an uninvited Fred suck my wheel. There are four options, aside from taking him down.
1. Drop him
2. Slow down to deliver a message
3. Stop and have a rest
4. Ask him to get off your wheel
There is potential harm, the wheelsucker could take you down. Different story with riders you know and accept the risk of riding together. Strangers with questionable riding ability? Not a risk I want to assume and it seems wrong to latch on without asking or getting an invite (hand wave from behind)