I moved up to 32mm and they have worked great for road riding and for the crushed decent gravel road and limestone/packet surface rail trails. For my old beater bike that I use on rougher trails (like the C&O Canal Towpath with is dirt plus roots, potholes, rocks etc.) I'm using 35mm tires and I've tried 38mm that one and bigger is better on those surfaces.
But for the 95% of my riding which is mostly road and just a bit of gravel and those nicely packed rail trails, 32mm is perfect and I weigh 230 lbs. I run the tire pressure at 80 PSI, down from from what I used to run. No meaningful increase in rolling resistance and definitely a decrease in flats over smaller tires/higher pressure.