I agree.
Even when I was training as a bodybuilder, I couldn't do pull ups ... IMO, that's a silly measurement.
I can cycle long distances, run reasonable distances, row, cross-country ski, snowshoe, hike up mountains, canoe, lift weights, do yoga, but because I can't do pull ups, I'm not physically fit??
Physical fitness is made up of 4 main components: cardiorespiratory endurance, muscle strength, muscle endurance, and flexibility.
I'm probably weakest in the "flexibility" category, but I do work on that from time to time.
I also think that there's an element of practicality that has to be considered. Can I do the things I want to do with my level of fitness? If not, then something needs to improve.
So for me, one of the things I need to do on a fairly regular basis is to walk longish distances carrying a moderately heavy backpack. It's a transportation necessity for me and I can do that. So in that regard, I'm fit enough for my practical purposes.