The wind does make a difference even if you're dressed appropriately, but not as much of an impact as if on bare skin. If you light up a thermal camera while fully dressed, the wind will suck heat away faster than on a calm day. Plus, if your clothing isn't taped together/nice and tight where stuff meets, like my wardrobe, there'll be a little more breeze on the inside. That does help keep the inner stuff a little drier though, so it's not all bad.
But I may not be qualified to speak. I only made it 80+ minutes yesterday in the same conditions.
I had a fender mount give way and after 100 feet of listening to the studs whir on the fender, I managed to temporarily jam the fender into an acceptable position that I wasn't sure was going to hold for the rest of the ride. It did, and new holes and zip ties have been added to that old fender for today's ride. It's a balmy 10F but that wind is gusting past 25mph. I'd take yesterday's 2F and 5mph wind over that...