Like anything, it varies depending on terrain.
The majority of marked MTB singletrack trails around me are pretty smooth and fast, mostly flat and a ton of fun to ride on a CX bike. There are sections where I have to slow down and pick careful lines, where on an MTB I'd just bomb through, but almost everything is rideable on a rigid bike. I've never understood why people in my area ride these trails on long travel bikes. Even when I ride MTB, a 100mm XC bike is plenty.
The same cannot be said for areas out West. There are many MTB trails in Colorado, for instance, that would not be fun on a CX/Gravel bike and even an XC style MTB would be challenging.
It's also funny to think back to the late 70's, early 80's when I was a kid. I routinely rode hiking trails around my house on a 10 speed road bike. My buddies and I would build dirt jumps and practice tail whips off of them, bomb through mud and gravel, set up log jumps, etc. I remember taking my 10-speed road bike into the LBS with a taco'ed rear wheel when I was around 12-13 years old and asking the shop owner if they had any wheels that wouldn't bend when I took them off of dirt jumps. He showed me a new Specialized Rockhopper with a triple front crank and cantilever brakes, and I knew I had to have one. I came home with a repaired rear wheel and a MTB magazine, and started pestering my parents immediately for a new bike.
