I commute everywhere by bike everyday, so my wardrobe has adapted to that. T-shirts, or whatever I'm wearing up top. Trunks or briefs on the pants yabbies. I rarely ever wear jeans, because they're pretty uncomfortable on the bike. I do have a couple pairs of cutoffs that are alright. Mostly I wear cut-off RedKaps or Dickies. I have a pair of Oregon Cyclewear Messenger Knicks that are stretchy and lightweight. They are the most comfortable piece of "normal" clothing I wear on the bike. I always wear my Sidi cycling shoes.
When I'm not riding to commute (just racking up miles with no destination except back home or possibly a coffee stop on the way) I love bibs and a jersey. I definitely don't feel like I "need" to wear "special" clothing on long rides, but I sure appreciate having something that is designed specifically for being on a bike. The added comfortability is immediately noticeable.