I rode my gravel bike on a 60-mile roadie group ride this last weekend. I had never been on these roads before, so I wanted to be prepared for any eventuality. I mounted 28mm Gatorskins on my spare set of wheels.
The other riders were all on dedicated road bikes -- S-Works Tarmacs & Roubaix, etc., etc. I never felt slowed by the gravel bike. In fact, on one rough stretch and on one 19% climb, I was glad I made the choice I did.
In a road race over smooth roads, is my gravel bike as fast as my road racing bike? No. But it's not far from it. And it's somewhat more comfortable over long distances.