Depends somewhat on the bike, a Crux or a Boone or something like that will work fine on the road; I know a few guys who ride those on the weekly "worlds" ride, sometimes with fat road tires, like 32s or whatever, and it does not slow them down. Something heavier and more bikepacking-oriented like a Fargo probably won't be as fast.
My Tricross works well as a road bike, but the 50/34 crank vs. the traditional 53/39 on my road bike tends to be a limiting factor, because I can't maintain a 100+ cadence for very long.