The 'motor' aside - Wind resistance is definitely the 800 lb gorilla in defining available speed... and of all the elements, the rider posture/position makes the biggest difference. So that would be the place for biggest gains also.
In another thread, someone posted a very interesting URL showing a comparison of different rider positions and the effects on wind resistance on power equivalency requirements - the primary idea of the article was to compare recent UCI banned positions - but its quite enlightening on the most popular rider positions in general...
UCI Banned Road Bike Positions Aerodynamics
clothing, of course, can have a great affect on wind resistance. There are marginal gains to be had with the bike, most are in setup - but at an expense when it comes to 'aero' gear..
Wheels/tires can have a very positive effect on overall speed, realized in increasing gains as one adds 'terrain' and climbing. It has been shown that fast descending doesn't compensate for the climbing time.
Lighter/faster wheels and faster tires also can have a great effect on rolling terrain. When one can power up a roller, maintaining a higher relative speed, the overall avg speed will rise quite a bit.
If one bogs down on short uphills, the overall ride speed can suffer greatly.
If I were to use one bike for both road and gravel riding; I would most certainly opt for a wheelset for each ride type. Changing tires whenever a decision comes up between road or gravel, would be a PITA. and settling for gearing which would be a poor compromise in either, would also bug me tremedously. I can;t imagine needing a 34,36, or 40 + in any road situation, including really long and steeper climbs... But I would certainly want closer gear ratios than my gravel cassette offers.
Ride On
Yuri