I think you need to also consider what's around you when you're riding, and how to handle a breakdown. I was riding last week and broke my chain. I was fine at that temperature, but I'd be concerned if it was so cold I wouldn't be able to use my hands effectively long enough to get back on my bike.