One drawback of mirrors, not a big one but still, is if it takes you an extra second to focus on the mirror and watch something strange going on you could miss a road hazard in front of you.
While it would probably never be an issue if you're going 10 or 15 mph, at 25 mph you could be running over something that was 50 to 75 feet ahead of you when you looked in the mirror. It's not the same as glancing in a car mirror. Those are stationary, steady, and easy to see. The bike mirror, whether it's on your handlebars or moves with your head, requires extra time to check. Extra time to see what's changed in the traffic (speed, direction etc). It can make a difference.
That's the biggest reason that handlebar mirrors on a drop bar just doesn't work for me. There is no way possible for the mirror to work both for the hoods and the drop position, without shifting my whole body. Craning my head. It takes extra time compared to my sunglass mirrors. I need to shift my head and focus so that takes extra time also, but not as much.