I think you aren't wearing enough black. Seriously, I only get honks or yells when I'm wearing a color. Apparently my black jersey has a subliminal message of "I will kick the **** out of you if you even get close to me." I like it.
If you already wear black, move.