If I know ahead of time that my route is going to have a lot of dirt and gravel, I'll take my CX bike, but if it's just a short cut on well packed dirt, there's no harm. In fact one of my (very few) KOMs is on a gravel levee that I got on my Tarmac. Just thought I'd take a short cut to save a couple miles, avoid the busier highway, and for funzies. Truthfully, a lot of the dirt roads in the Delta are smoother than the "paved" roads.