I think I have a memory of tight shoulders and neck stiffness when I first started riding gravel. I was riding 700x40c tires, carbon frame, carbon seat post. And too much air in the tires.
I discovered that less air pressure in a bigger tire meant the tires would absorb the shock, eliminate the bounce and give more traction and control. I now use a 29x3 plus bike on gravel (@ 13-15 psi) and the older adventure bike has the same set of 40mm touring tires I started with, for paved roads.
All considered, my plus bike is my favorite ride on gravel or easy singletrack, though gravel forest service roads and old double track are what I really enjoy. I feel no stiffness, no soreness, but lots of confidence at speed in the downhill corners.
Alternatively, your riding position may have your hands too low with respect to the saddle, which can give neck and shoulder soreness also. An adjustable stem that rotates upward might be a partial answer.