I started with a gravelish bike in the Surly Midnight Special (steel, 650b x 47), rode it for a couple of months and recently bought a Specialized Tarmac (carbon, 700cx26mm) and just wrapped up a 35 miler today. I can tell you what I feel the tradeoffs are:
Gravel bike = versatile, comfortable, smooth over rough paved road, fairly stable handling, lots of fun and comfort, but...gearing is not optimal for road speed (at least not what I have), and wheels/tires feel slower off the line (stop signs and red lights). Overall, feel like I can't get up to and maintain a good as speed easily.
Road bike = faster, quicker off the line, sharper handling, better gearing for road riding, feel like I can maintain a higher speed more easily, but...not quite as comfortable, feel the rough paved sections, less stable (I am getting acclimated).
Overall, I am happy I got the dedicated road bike for road riding workouts. I am also happy to have my Surly for cruising around with the kids, going on gravel trails, and more urban type cycling. I thought the gravel would do it all, but I was wrong.