Back in the Dark Ages (aka, the 1990a) , I put a pair of skinny slicks, Spinacci bars, a rear rack, and panniers on a XC bike and road with a buddy fron Oakland to LA along the PCH. He waa on a touring bike and I was just as fast. The front end was a little twitchier than a road bike but I got used to it and could pound out the miles in an aero tuck with the Spinaccis. So you can go quickly on a XC with proper tire choice. But a couple of things to note...
1-the longer crank arms of a mountain bike will affect your cadence.
2-you'll likely be geared lower than a gravel bike, so you'll climb better but might run out of gear inches on paved descents.
So whether or not you should get an XC depends on just how chunky the gravel you want to ride is. If it's dirt farm roads then a gravel bike may be the right tool. But if it may include some single or zketchy double track, then XC with the right tires may be the way to go.