Eh. Proper road bike clothes are really a boon for long distance riding, not just because of padding (after all, trisuits have barely any but are nevertheless pretty comfortable).
That said, if I'm running errands on the bike, commuting or trying to squeeze in a quick easy ride sometimes I'll just get going in everyday clothes. You can get by even long distance riding in them, but why would you? A lycra roadsuit makes you faster, is more comfortable, is practical due to pockets, and just looks right and good on a road bike, especially if you aren't overweight.
Wear what you're comfortable with, but no need to shy away from silly "cyclist" clothing 😉