I kinda agree, you're being a bit 'wimpy' at those temps, but there are people who are very sensitive to cooler temps, so take that statement tongue-in-cheek.
Try different combinations, and don't skimp on quality. For my head, skull caps work to below 40. I have no use for vests, as my arms are always as cold as my torso. I like Craft base layers, and my best discovery a few years ago was that they made some base layers that had wind protection on the front side. And don't neglect the neck.
My weakness on cold rides is almost always my feet. I also discovered toe warmers a few years back, and they work for me to a bit below 50. I also invested in some winter shoes, but I didn't get a big enough size so they don't do the job. At freezing temps, my feet are miserable after 1.5 hours. I dislike wearing full booties.
Follow the common advice to dress is layers of breathable material. Remember that when starting out on a ride in cool temps, if you're not cold starting out you're over-dressed.