You don't just ride up behind someone and sit there without saying anything. That's a big violation of cycling etiquette. If you're going to sit back there, at least let him know you're there. Say you pull up behind someone and it's a windy day or that person has bad hearing or something and doesn't know you're there. Bad things could happen in a situation like that.
You say you were far behind him and not drafting, but how is he to know that? I doubt he was monitoring how far away you were, so if he heard you back there he probably just assumed you were back there drafting and not helping. What if you thought someone was drafting off of you for a few miles? You'd probably be a little upset about it as well.
If you caught up to me like that the 2nd time just to make me mad I'd start blowing snot rockets.