Anyone who says cold slows them down more has clearly never ridden in real heat.
You can mitigate cold by dressing warm. You can't do the same for heat because you can't get more naked than naked.
When the air temperature is warmer than body temperature, the wind is warming you up. Only evaporation cools you at that point. Performance tanks as you enter the early phase of heat stroke.