I ride mostly carbon rims, but I don't think they are magic. They may be lighter, if not too deep. Depth is an issue with crosswinds. I'm in fact getting ready to build a shallower 28 mm depth carbon wheel just to put on the front when the forecast is for higher winds. I normally run 45 mm depth. Of course, you can go a lot deeper than that.
I'm not sure you would feel much difference between what you have and carbon wheels. Potentially lighter weight which you may not notice. Stiffness, may be a thing, but it doesn't sound like you are pushing the limits of wheel stiffness like racing might do. Or if you went deep on new rims, weight may not be much different but you might be more aero.
I'm 62, not a racer, and just like nice things. I like the sound of my medium deep 45 mm rims.
On my main bike (A Trek Checkpoint gravel bike), I only recently got around to building up the set of 45 deep wheels for road riding. I specifically chose them for their external width of 32 mm. It pairs very well with a Continental GP 5000 28 mm tire (30 mm actual on these relatively wide rims). Gives me a near optimal tire to rim width ratio for best aero performance. Can I tell the difference? Probably not. What I can tell the difference is the tires themselves as compared to the 40 mm wide gravel tires.
I picked up 2 MPH for the same effort. Absolutely huge difference in rolling resistance. I'm saving 40+ W, according to bicycleingrollingresisatnce.com. Now, this is a very good road tire (Conti GP 5000 S TR 28 mm) vs a mediocre gravel tire (Bontrager GR1 Team issue 40 mm). So, very much an apples to oranges to oranges comparison.
Point is, if you want to feel a difference, compare the rolling resistance of the tires you use against the best you could use. You might be surprised how big a difference that makes, unless you are already using a high performance tire.
To the OP, if you just want to ride something different, then I say go for the carbon wheels. But I'd get something that would be more different than what you already have. Not just material. If your current rims are shallow, then get some medium deep carbon rims. If it's going to be windy, you can always put the old, shallow front wheel on.