Or ... dress for when you are warmed up. Stuff a newspaper (one folded sheet so 4 pages) under your outermost jersey from below your waist up over your shoulders, ripping a few inches down the center and fold open like a polo shirt. Best p[art. when you warm up, pulling it out and folding or just stuffing into your jersey pocket is easy. And if the weather turns, it can be a ride-saver.
If you can find film/video footage of the rest of the European pro peloton cresting a mountain pass, you will see racers inserting that newspaper to this day like they did 100 years ago. No, not the flashy team leaders. They get handed up real jackets. But their nameless domestiques who have already done their work for the day and are minutes back but have to finish to start tomorrow's stage - there's no one around to hand them a jacket. Getting too cold on the descent could mean getting sick and not being able to support that team leader (and potentially, not getting a renewed or better contract next year. So it's newspaper. Takes almost no jersey space, light, cheap, easy - both on and off - and works really well.
Granted, you won't impress anybody.