My wife is a long-time practitioner of a hot yoga discipline known as Bikram, so named because of its founder. This is a specific emphasis on spinal health and the strength and flexibility necessary for optimum spine care. At any rate, there's a type of clothing that all of the Bikram people gravitate toward eventually...and it has the following characteristics.
- lightweight
- form fitting, and minimal skin coverage
- transmits sweat away from the skin
- non-absorbing, quick drying
- easy care, durable for frequent washing
I offer all of this because it has a parallel to how long-time cyclists come to understand their cycling gear. Can one take a Bikram class in basketball shorts and a cotton tee shirt? Certainly, one could. Would there be additional discomfort because of baggy sweat-soaked clothes? Yes, and this may or may not be perceptible to them.
I've been riding a long, long time. Can I ride wearing anything? Sure, I can. But I've found (by decades of trial and error) that I'm most comfortable wearing cycling specific garments for each season of the year. They simply work best for me.
As for "padding", I'm never a fan of it. Not for handlebars, not for gloves, not for saddles and not for my bottom. I wear stuff that is optimized for me. If the OP doesn't want or need padding, then maybe tri-shorts would be the way to go. Thin, minimal friction protection that's not only well suited to immersion/quick drying but also to high intensity riding. Just a thought.