Really you just have to worry about how much clearance you want between the tires and frame or where ever they will rub first. It's really a personal preference thing and what kind of surface you ride, IMO.
Some wouldn't feel comfortable for their safety with any less than a ¼" . Some might feel comfortable with whatever size won't rub.
If you ride in mud sometimes, then you might want to have more clearance. The gumbo mud around here can stick to your tires and build up even beyond a ¼"..
I tend not to worry too much about rim size because IMO, you'll probably max out your tire size with the frame before you max out the recommended max tire width for that rim. Unless perhaps you have a very old bike from the days of 18 mm tires and even then I doubt you'd exceed the max width rec. first.