I have far less time than I used to for road riding - son, wife, full-time job. At this point, because my fitness (and waistline!) are not where they should be, I definitely prefer riding a more relaxed geometry. My body just isn't use to the positioning you have on a pure road bike right now.
My favorite bike to ride these days is my Van Dessel Gin & Trombones cross bike with 40mm Continental gravel tires. Still plenty fast on road and gravel, but more upright position and not as stretched out. If I had to downsize my bicycle collection, I'd get a nice cross bike with two sets of tires - road-ish ones, and off road ones.