I wear bike specific stuff on my commute. It's 8+ miles so 30 minutes or so when pushing hard. Living in Phoenix also means that about 8 months of the year my morning commute is over 70 degrees, and 4 months of the year it is close to 90 degrees. Wearing work clothes wouldn't make sense since I would have to shower anyway.
In the summer I wear lycra shorts and either a jersey or a technical T-shirt, winter it will be either tights or mtb shorts sometimes with leg warmers, tech T, and arm warmers or jacket.
Even if sweat wasn't an issue, I find most non-riding pants/shorts to be uncomfortable on the seat. The seams just aren't in the best places and with the effort to find non-cycling clothes that are comfortable to ride in I may as well just go with cycling clothes.