Depends on the brake design. Some will nudge aside easily, others tend to drag more heavily. If your brakes have an adjustable opening (rather than simple open/closed quick release for wheel changes), you can experiment. Ride awhile with the brakes rubbing. Then open the calipers so there's no chance of brake rub (but still have at least some braking) and ride awhile longer. Switch back and forth, see if you notice any real difference.
But it may have just been a bad day, for all the reasons so colorfully described above.
In my experience, all things being otherwise equal, I can usually attribute riding slower to wind resistance. I don't have any aero kit or skin suits, but I do notice a significant difference between my snug fitting jerseys and t-shirt-fit casual jerseys, especially if there's any headwind -- even 5 mph is enough to make some difference. Or if my back, neck and shoulders are more stiff and achy than usual, I'm probably sitting more upright. And if that stuff is stiff and achy, my ergonomics are probably terrible and I'm just not able to put out the usual power and smooth cadence. My ortho doc, PT doc and chiropractor have all commented on my injuries, terrible skeletal alignment and cyclist posture, so I'm working on it. They're trying to make me "better." I'm just trying to get faster.