No, no damage. It's more likely that eating less or even nothing on shorter rides improves your ability to do longer rides, by forcing your body to improve transfer and use of fatty acids, also known as fat burning. The issue on Imperial centuries and longer rides is that you'll run out of fast-burning fuel: carbs in the form of glycogen. One can't climb well once the glycogen is gone. Fueling right from the start and keeping it coming is the key to doing long rides because that spares glycogen. Long distance riders don't even consider 100 miles a long ride. That's maybe the lower edge of the spectrum of long rides. IME the first 3 hours of a long ride are the most critical. Fueling right from the start initiates the correct pattern for the whole ride, and starts sparing glycogen right away.
Thus the idea of improving fat burning by eating less or nothing on short rides is also a method of glycogen sparing for long rides, by getting your body to substitute fat for carbs at the lower end of effort.
__________________
Results matter