After years of wearing merino wool and other high-tec materials and still getting too sweaty from overdressing, I now pretty much just use regular clothes - including khaki pants, wind pants (like the adidas swish pants), a skull cap, gloves, and various t-shirt combos...even COTTON...yes, I went there.
I once work a cotton waffle layer and a t-shirt, gloves, and skull cap in the 30's and felt completely fine....cold at first, so no sweat on the cotton.
Cotton sucks when it is wet from perspiration, but if you don't over do it you're fine IMO. As a bonus it doesn't stink like polyester and polypro.
If I lived where it was below 0F and was out riding for a long time, I would probably avoid it though.