I had been riding in sandals and ordinary khaki shorts for years. Rivendell influence, I guess.
I tried to go for biggest improvements first, so I started with the shoes, then the shorts, then the clipless pedals, and then the jersey. Each made a notable improvement. And each improvement was subtler than the last. In the end, I think the jersey makes the least difference and may even be worse than a nice linen shirt on a hot day going 15mph or slower.
I still commute in sandals and socks and ordinary clothes, but right now it's only 2 miles. I'd probably start with shoes, and work my way up the list, if my commute became any longer.
__________________
Owner & co-founder, Cycles René Hubris. Unfortunately attaching questionable braze-ons to perfectly good frames since about 2015. With style.