I don't think "are gravel bikes slower on gravel" is the question.
I think we've all put up with a 40mm or more tire on pavement to get to the fun stuff. Sometimes it feels like a tractor. Other times I think, wow, this isn't terrible. I never think, while riding a gravel bike on the road, that I'm faster than if I was on the road bike.
I don't get my wife to follow me and switch bikes. I don't think she would if I asked. Too bad, I've got one I'll do soon that would be great for gravel, road, mountain, then road bike. In that order.