I think I would have given the "what the heck shrug" if he was yelling at me. Personally I would like to know where all the hostility is coming from. Usually I just ignore them since they pipe down at a light.
A lot of these guys are tough guys in their vehicles, but won't step outside of them if they really have a problem. Of course they probably don't realize i'm 5'9 215 (high of 240) built like a refrigerator while i'm riding. They probably think we are all Lance look-a-likes.