Unless the rust is more than surface rust, and from your description that's not an issue, it is well worth the time to restore...
Of course that is answering your title, the text seems to read like you are going to restore, you are just looking for advice on removing rust without damaging the paint... There I can't help you.
