For more than 40 years I've never kept the caps or the nut that Presta valves come with on the valves, and never had a problem. I did have issues with a couple of cheap valves found on Slime tubes I used once. However, on my touring bike I leave the cap on since that bike will see a lot more rain and dirt, so I felt I would rather be safe than sorry while far from home.
Anyway, all Presta tubes come with the cap and nut, you don't need to keep the old ones unless they're some specific color you like you can't find on the new tube.