I don't think I could go back to riding a bike with cantilevers as my main bike bombing down steep descents and dirt B-roads. Looking back and what I did on a bike with crummy brakes, I am honestly pretty shocked things didn't end badly at least once.
Like you, I also do gravel races for fun, and to have a goal to maintain a good level of fitness instead of lollygagging all the time during adventure type rides. I think it's just common sense to have maximum control of your bike if you ride in the bunches that happen at these events, and also being realistic about not taking it easy on every decent if you don't have confidence in your brakes.