within reason: leave the residue was what my master teacher taught me, it allows a better bond for the next layer of glue, and a few layers is nothing to sweat
if..however, you have a ton, yes, then lightly use wire brush or lightly sand down carefully and then leave appropriate amount to bond to
It's actually easier regluing than new rims that need a couple layers applied and dry time between