I follow a lot of different levels of riders on Strava, the fast ones put in a ton of time/effort. Like riding everyday, on a trainer or outside. Doing structured routines like intervals, weightlifting. These guys are just cat 3 and up probably, weekend warrior racers. I've thought I definitely need to do some weightlifting myself.
I noticed you said you don't run tubeless? Tire choice and pressure will make a bigger impact on gravel. I run tubeless wtb riddler's, 37mm at 35/35. I'm 180lbs. I'm pretty much the same speed you are, a touch faster but with half the climbing.