It depends ...
I have no problem with another rider on my wheel if it is a temporary situation. For example, if they come up behind me and they're planning to make a turn within a couple hundred metres, there's probably no point to passing me and then making the turn ... unless they're really fast, of course. Or if there's lots of traffic or something and it is dangerous to pass, then I have no problem with them being back there.
But if a rider attaches him/herself to my wheel for longer periods of time ... that's just creepy and potentially dangerous.
Creepy because really, how many of us want someone in our space for long periods of time. Think how you'd feel if you were walking alone on a path and all of a sudden someone started walking right directly behind you. Maybe it wouldn't bother guys, but as a woman, I'd be completely freaked out by that.
Potentially dangerous because I forget to point out things to people behind me at the best of times! I may also suddenly stop to take a photo because I've seen something right there I want to capture on camera. At times, I spot side roads I want to explore and have been known to slow suddenly and turn. And if I'm climbing, all bets are off. There's a really good chance that if that hill gets a bit too steep for me, I'm off my bicycle and walking ... and there will be no warning. If you leave a gap of several metres between me and you, you'll be fine ... but if you're right there behind me, it's you who will be going down if you hit my rear wheel.