I prefer to make sure my jersey is visually striking. Some roadie kits are dark, some are bright. I prefer to be visible.
People's skin sensitivity varies. I'm one of those who needs to get out of the shorts and clean down there as soon as possible, and wear clean shorts every day. Saddle sores are no fun, and they've ended some people's tours. Ergo, I take off-bike clothes.
But on the bike? For hours of daily cycling, I can think of no reason to wear anything less than bike shorts. When you go into a store, diner, etc., look people in the eye. They'll look back.