My feeling is that you should hang back or pass. I know I don't like a stranger hopping on my wheel, and when it happens (and it does occasionally) I either slow down so they'll pass or I try to drop them on a climb. If neither works, I'll pull over and stop for a bit.