So what happens when tire ages and stretches 0.5mm? What happens when wheel loses perfect true? What happens when you hit a pothole? What happens when you get in shape and find a bigger kick and the frame flexes more than usual? What happens when tire picks up a piece of sand? A tiny pebble stuck in tread? Why are you living all the way on the edge? Is this how you want to go?
If someone knowledgable in carbon layup tells you there is still a frame there the only recourse is to go find some undersize tires. You're already on 23s. 700x18 is mostly dead stock antiques.