It's more about the gear and getting acclimated. Mindset is also important. I've had people tell me I'm crazy for riding a bicycle on good roads below zero F but when I ask them what they are doing on the weekend they'll tell me "skiing, snowboarding, or playing hockey" on an outdoor rink. Same temperatures, similar amount of physical activity, similar speeds but I'm nuts
.
There are plenty of threads on how to select and layer clothing for cold weather. Properly outfitted in clothing and bike, you can easily ride in temperatures well into negative digits. One tip is to get a ski helmet. They are designed for similar impacts and similar speeds to cycling but have fewer vents and built-in insulation and a clip for ski goggles. Wool and appropriate synthetics are your friends and cotton is evil.
Don't expect to ride at the same speeds and position as you will during the summer, but you can still get in a good workout and have a lot of fun.