Another vote for AmFib bibs. Several years ago I scored some unpadded ones for relatively cheap, $60 or so. I was hesitant to order unpadded, but it works out for layering when needed. I wear a padded short du jour underneath for 20-30F. Below 20F and I add in a set of thick PI leg warmers to the shorts, or a full, padded Craft bib under the AmFibs if I am worried about dropping even lower. I also use Lake MZ303 boots for anything below 45, since my feet get cold regardless of temperature. The Lakes are nice, but not perfect. Double socks (Costco’s Smartwool knockoffs are my usual daily sock) with a thin Smartwool base sock. For really cold temps I will remove the insoles and wear a neoprene dive bootie over socks. Above 45 and I wear my usual Shimano touring last shoes and an old Cannondale neoprene cover (well worn and full of shoe goo, I might add).
I try to have the bike fully loaded and ready to go before dressing. Still get hot, but I save upper body clothing and helmet for right at the door or even outside.