I wear matchy clothes when I race or I'm on a group ride. (The nice people who give us discounts and swag in return for wearing their logos really appreciate it.) But, I ride pretty much every day and laundry just can't keep up, so on solo rides or commutes, I'm whatever's clean, usually a mishmash of prior seasons' kits.
But you bring up a good point: you can usually tell a rider's ability from a few pedal strokes. Is he balanced? Are his elbows and shoulders loose? Can he look back or take a drink without swerving? Nice easy spin? Stuff like that.