Patience is the key. I don't have the ultra riding experience you do nor have i suffered the same setbacks but my riding, my speed, and my endurance dropped off a lot during my last 2 or 3 years on the job. It got to the point where I was pricing e-bikes and checking their range to see if I could do group rides with my friends. I also considered switching back to motorcycles.
Instead, I retired when I turned 65 and spent months riding 4 days per week and I actually started hanging on to the faster groups and sometimes I wasn't last. It takes time and patience. It's frustrating and even after seeing obvious gains sometimes I want to give up, at least give up keeping up with younger and/or fitter riders.