Lack of rideable pavement in short. In my area, the paved roads tend to have a 55mph speed limit and no shoulder. Or, at least not a paved one. If there is a shoulder, it's gravel and the pavement in the right 1/3 of the lane is likely broken and potholed anyways. Even 28mm tires aren't going to fix that... I love road cycling first, but sticking to pavement would mean limited route options and a lack of safety. Gravel bikes (which to me is just a road bike with fat tires) open up basically every road in the county, from flat to hilly and with great scenery as well. I can safely ride down a gravel shoulder of a high speed road if need be, but usually ride the dirt roads. Technically the dirt roads also have a 55mph limit, but the traffic volume is so much lower that drivers almost never have a problem giving me a comfortable passing space.
In the end, it's still road biking and in my minds eye I can look like one of those properly hard guys from the early days of the Tour de France!