I think it's a learning curve as you try to flex a bike made for one genre into another. If you have a bike optimally set up for gravel it's not going to be optimal for more technical terrain - you would do things differently for that.
If the seat is set for speed on gravel roads it will be too high for control on single track where you want it lower. A good argument for a dropper on a gravel bike I suppose. When I started riding my FG more on gravel I had some problems with the old school narrow drops. The hand position was too close together to negotiate some detours that go around gates and loose gravel on descents.
So I swapped them out for flipped utility bars, like very shallow dirt drops. Wider grip and angled for better control.
I still can't do wheelies with it either.