Depends how you define fitness.
In theory you could find a 400 pound guy with a 400w FTP. For the same of discussion. We'd call that guy pretty fit because he can sustain a big effort. We'd also drop him on every hill. Like his w:kg, the guy's VO2max would be pretty low, and that's a way to describe fitness too. VO2 won't tell you who's going to win a race, but it's an important health metric; you're more likely to be alive in 5 yesterday if you're on dialysis than if you have a very low aerobic capacity normalized by weight.
But the shorter, down to earth answer is that getting fat generally means less time exercising, and that's where the sluggish feeling comes from.