I've stopped riding gravel because it can rattle my brain. I'm a TBI survivor. Yes, I can pick my gravel more carefully, go slower, use bigger tires and low pressure, etc. but I'm also an ex-racer. Going slow doesn't always happen. Plus gravel roads can have surprises like severe washboard at the bottoms of hills too steep to ride slow.
So, I'm accepting that my NFL lineman's battered brain is far better served riding hard surfaces.