Mostly other cyclists annoy me. Those that run stop signs and/or red lights. Also those that cut to the front of the line at a red light, then ride really slowly.
Also, as someone else said, people who use those extenda-leash things with their dogs. They're actually illegal in my city (Toronto) but that doesn't stop anyone.