What you want is a CX racing bike. Giant TCX, Cannondale SuperX or CAADX, Trek Boone, etc. are light and nimble, and perfectly at home on pavement and groomed gravel. They probably won't fit anything bigger than a 35mm tire (actual), but that should be plenty for the kinds of gravel you're talking about. Since 1x is the go-to configuration for CX bikes, you may want to consider having your shop upgrade to a compact double to tighten up the spacing for fast group rides on the road. Give one a test ride and I think you'll quickly feel the difference between CX-racing and gravel geometry.