Welcome to my world. All my rides involve some portion of trail surfaces with large and small crushed stone, stone dust, dirt, sand, rock, and just about everything else. I suppose there is a bit of increased drivetrain wear, but I’m still replacing chains at 3000 miles or so. So it’s fine.
Do watch for hitting a sudden patch of very loose sand or similar material when cruising on firmer surfaces at speed. It’s like someone slammed on your front brake without letting you know in advance.
Otto