If you can afford new wheels (and TIRES) and they allow you to drop several hundred grams, I say go for it. Depending on the tires and tubes, you can save a couple hundred grams just there as well. Key concepts being "afford", "several hundred" and "tires and tubes".
I'm a marginal weight weenie, but I noticed and enjoy lighter wheels and light good quality tires and tubes (again, assuming the difference is several hundred grams, not just a hundred or so). I don't think I go "faster" in any meaningful way, or at least that makes any meaningful difference in a 30 mile ride with my buddies. Can anyone tell if they're even 30 seconds faster on any given ride? I don't fool myself in meaningful performance gains. But, to me they just feel livelier accelerating - which is just fun. And the whole bikes feels lighter when i schlep it into the back of my car or on to the hooks in the ceiling of the garage. That's also fun.