Because drag increases with the square of velocity increase, marginal differences in the aerodynamics of the bike will have more benefit for fast riders than slow. Obviously, going from a hybrid frame to a TT frame is more than a marginal difference, but if your cruising speed is, say, 15 mph, you probably aren't going to see a whole lot of difference going to aero wheels, etc.
Obviously, the rider is the least aerodynamic element on the bike, which is why enclosing the rider in cowling is so much more effective at reducing drag than anything allowed by UCI or, for that matter, anything practical on the road.
Cowling a recumbent rider:
https://qz.com/818769/the-worlds-fas...-go-at-89-mph/