Basic differences between a gravel bike, and a road racing bike are going to be: 1) tire width; 2)geometry; 3) position, 4) weight, 5) gearing.
Tire width is obviously manageable just by putting on narrower tires.
Geometry may make the GG a little slower handling, but likely hardly noticeable, and totally manageable.
Position. The typical gravel grinder is likely to more naturally set up in an upright position, and therefore a bit less aero. This is manageable by moving spacers, flipping stems, or at the most simple, just bending your elbows.
Gearing. If you spin a decent cadence, lower gearing on most GG's isn't going to hold you back, can be managed by swapping rings or cassettes, and the lower gears will likely be an advantage climbing.
Weight. The couple of pounds heavier for the typical GG will be a minor disadvantage, but when you do the math fairly nominal.
Back before we got all specialized, a bike that had drop bars, relaxed geometry, clearance for wider tires and fenders, and wide range gearing, was called a bike. As bikes have become more specialized, we've renamed what would have been considered just a regular bike a Gravel Grinder.
No reason not to ride it on the road.
__________________
You could fall off a cliff and die.
You could get lost and die.
You could hit a tree and die.
OR YOU COULD STAY HOME AND FALL OFF THE COUCH AND DIE.