Instead of a towel or "camp towel" (those blue thingys), go to Wally-World and get an $8 man made chammy in the auto department. Take it into the shower (or whatever you are using), rinse it out, hang it within reach, shower, use the chammy, wring it out (marvel at how much water it absorbed), and put it up wet. It packs into a small cylinder, packs well and doesn't need to dry out.
Now you aren't going to be absolutely dry when you use this thing, but in 30 sec you will be.