Convertible jacket
My older Pearl Izumi convertible jacket works great. The sleeves zip off, connected by a shoulder yoke. There's a large mesh panel at the top of the back side of the vest, venting under the yoke. So I can pull the front zip halfway down and get a big airflow through the jacket on the climbs. I can avoid getting too hot at those slow speeds. And the vest itself works great when it warms up during the ride. The fabric is quite wind resistant, but allows just enough air through to keep me from getting too damp from sweat.
Unfortunately, PI has redesigned it to be less usable, and it's quite expensive. I'd look for similarly designed jackets from other sources.
Thin base layers
I use this jacket on my coldest rides, with more base layers. I used to have some heavy base layers, but now I just use 1, 2, or 3 very thin baselayers. I have REI labeled ones, including one with a zip turtleneck, and two Patagonia long sleeve base layers--those are pricey, but last forever, and are amazingly light and packable. It's surprising to me that two extremely thin base layers do a good job of wind blocking on moderately cool days, and work great as insulation under the jacket when it's colder. Two (or three) are way better than one thicker one, and much more adaptable to different conditions.