Delicate?
Its more of the sweat thing. Shorts and jerseys get wet from our sweat. And depending on the humidity, clothing can get very wet. Also, the shorts have the padding, more or less. That padding has to absorb all that sweat as well.
To preserve the life of the material, it seems to me that the shorts/jersey needs to be rinsed out. No matter whether you go in the shower, or hand wash, or throw in the washer, the material needs some kind of rinse to remove the sweat.
What happens when you leave the sweaty shorts/jersey on the floor or your hamper, waiting for tomorrow's laundry day? Its going to dry out, sweat included.
Will that dry sweat have some kind of effect on the material? Maybe.
Now the "waste of time", "other better things to do". Like wiping down the bike after a ride? Or cutting your toe nails that get too long that they begin to cut through your socks?