The surface (in my experience) is not the cause. It is biting off more than I should chew, so to speak. Going big, exceeding my conditioning, and bam, sore back.
Riding a bike is not a non-strenuous activity unless you plod along at a low power outputs, use an e-bike, and/or avoid things that make you work, like hills or gnarly surface routes. Honestly, though, when I get the sore lower back, it is usually on pavement because I will push a lot harder there than on surfaces that are not as surefooted.
But my back is not your back.