There is a lot of grit, dust and sand on the paved roads I ride. So if riding in your gravel causes more wear and tear, it's probably not much. Possibly more wear and tear will come from simply the more power that is required to be handled by your drivetrain to ride on those conditions.
Now if you are rolling around on the ground with your bike trying to get the fleas off of you then I suppose there is a good chance you'll get more grit in the bearings.