I've got a Domane SL6, a great road bike and I have ridden it (with 32mm tires) on gravel stretches of local rides - the bike is fine on the gravel stretches, the tires (Continental GP5000s) are too narrow and too slick.. With 38mm tires with the right tread, it would be fine for those stretches. More than fine - the ISOspeed damping is really nice for small bumpy stuff..
But, I also do touring and some riding on towpath or 90% gravel rides and I ended up getting rid of two older bikes (an old hybrid beater bike I would use on dirt/gravel/towpaths, and my 25 year old Trek 520 touring bike). I weigh 225 and the wheels on any good road bike aren't made for my weight plus baggage on touring or extended use on rougher stuff. I picked up a Jamis Renegade, just did a few days touring in Florida on it (CycleBlaze Journal
here) and really like that bike for touring and mostly unpaved rides, love my Domane for the 95% paved rides.
If I weighed less, I would have just stuck with the Domane for road riding and touring, and the old beater bike for the rough stuff. But, by going from 3 bikes to 2, I was able to justify buying the Jamis - doing an N-2+1!