For pavement and crushed packed stone rail trails, a tire that's mid30s is fine. So a 40mm tire is definitely fine.
For gravel, it totally depends on the time of year and where you happen to ride, but a 40mm tire will be plenty for 95% of the time.
I rode 40mm gravel tires for 3 years in Iowa, Missouri, Nebraska, Illinois, and Colorado. It was fine for almost every mile of riding.
The only time I wished for a wider tire was when I would come upon half a mile of freshly laid gravel and when that happens, pretty much anything smaller than a fat tire will limit your speed.