My cold tolerance keeps getting worse with age (62 now). A few years ago I'd happily ride with temps in the 20s. I'd join casual group rides in any weather, even early morning rides -- and I'm definitely not a morning person.
But the past couple of years I've battled chronic sinus inflammation and cold dry air gives me severe headaches. I tried various face coverings -- paper surgical masks, balaclavas, microfiber mufflers -- and nothing helps. It ain't worth the hours of misery that follow an hour or two in the chill.
So now I won't ride if it's below 50. If it starts warmer, as soon as it drops I head home. The only exception is damp weather. I actually enjoy the chilly rides in damp weather, as long as I'm wearing the right clothes -- rain gear or at least wind-resistant gear so my sweat doesn't get wind chilled. The cool damp air actually loosens up my sinuses and my head feels clear, although a lot of mucus expulsion is involved.
I had planned to join a casual group ride this morning with friends I haven't seen for several weeks, but when I checked the weather it'll be in the 30s-40s until noon. So I wimped out. But I'll ride later this afternoon after it's in the 50s.