Every concession a gravel bike makes in order to improve off-road performance, will negatively impact its performance on the road. So you need to ask yourself which kind of riding is most important to you. You can't have the best of both worlds, but you can decide which kind of riding matters most to you, and then make accommodations.