Only when I am on it. But (a) I am also 20% slower than most people on my road bike, and (b) my gravel bike is my road bike.
Often it is slower to ride on loose gravel and steep, rutted inclines than it is to ride on smooth, well-maintained roads with more moderate grading.
If your gravel bike slows you down on the road, you probably just need more supple tires.