A gravel bike is a great bike for a ................gravel road.
You have stated that you will have 2 sets of wheels, so the main barrier is the gearing on each bike is not suited well for the other use. I would try to find a bike with a triple on it. That might not be easy now. For your application the triple could solve your problems. Maybe look for a touring bike that can take big tires. I'm not endorsing this bike in any way, shape or form. Something like it though, that can take big tires would do the job.
Save up to 60% off new Road Bikes - Motobecane Grand Record