I don't feel it affects my ride. You're quickly back up to speed. It's more about the trouble - so it has to be a shot worth taking, unique in a way.
I tend to take more generic shots in a short pause for eating or resting, just to have some documentation of that particular ride.