Different people have different ideas as to what constitutes being fit.
To be able to compete in the Tour De France, or even to keep up with those very fit riders, most of us will never do it.
But to be fit enough to get around on a bike with ease, is a realistic goal for most people.
If you try to keep fit when you can, then go through a time when you exercise less for any reason, to get back to fitness is probably easier than if you never tried to get fit in the first place.