This is my experience going from 28-32-38 tires. 28's roll great and work well for lighter loads around 20lbs on the rear rack. Commuted for 10 years a 30 mile round trip in all weather including snow. Gravel roads as long as not slick with heavy mud were no problem.
32mm will carry a standard 35-40 pound load without trouble and roll well. Long tours on them were just fine, and they can probably handle even more weight, but never needed to try as I travel light. Commuted a 5 years on them before going to 28mm tires.
38mm tires handle a heavier load, and I have had up to 50 pounds on the rear rack without any issues. This is what I currently have on my touring bike. Without a load they roll OK, not super fast, but they are OK for commuting and touring. The extra weight of the 38 over the 28 is immediately noticeable when accelerating, including climbing. Between 32 and 38 it is less noticeable when climbing, but does show up when accelerating.
When the current pair of 38's wear out I will replace them with a set of 32's as it is my preference to push as little weight as possible.