Unless you do serious single track riding, road tires are definitely worth having even if you have only one wheel set. I think you will immediately appreciate the difference. With a spare pair of rims, it's a no brainer.
What kind of brakes do you have? If you have rim brakes, check the rim widths of your two wheel sets at the brake track. If there is very much difference, you may need to make a brake adjustment when you swap wheels.
__________________
My greatest fear is all of my kids standing around my coffin and talking about "how sensible" dad was.