In the early days of my Pete Mooney, the rear rack came on and off as needed, Year 5, I had a replacement fork made with braze-ons for a Lowrider which also came on and off, Now the swaps are far less often and the bike has not toured for many years but it spent the '00s and first half of the teens as a farmers market bike so those racks saw real use.
With carefully done threading and care starting bolts, those dropout eyes should handle the changes many many times. And when the threads do go, a slightly longer bolt and nut works just fine on most bikes. I've had bikes that came without threading and older bikes that had them no longer. Small PITA but far from a frame killer.
Oh , my experience is all with steel and titanium frames, I've never owned either aluminum or carbon fiber. (The ti frames have only seen fenders, not racks but the fenders come on and off a lot.