I had one cut off my hand about 9 years ago. In general, it is not something to panic about, but about 10% can go metastatic (unlike basal cell), so you definitely want to be rid of it. I avoid peak UV and wear long-sleeve jerseys and use zinc sunblock rather scrupulously.
Dermatologists can be a bit dim-witted. Mine, for example, told me to wear sunblock when flying in an airplane to protect me from cosmic rays. They tend to give the same priority to someone with pimples on their face.