I can't handle the cool temps and get stiff too. Pack a backpack or rack back with a spare long sleeve jersey or maybe a lightweight warmup suit to slip in to. Duck in the restroom, strip out of the wet jersey and wipe off with paper towels and slip into the sweats / long sleeve jersey. Walk back to the bike and pull the wet jersey over your handlebars to dry out during the meal. Go and enjoy your company. Maybe order toast or something with an OJ if you get dirty looks for not ordering anything.