To me, sloping top tube frames are more nimble climbers. When out of the saddle the top tube can sway lower down the legs (maybe even below the knees? I’d have to check this). Certainly more to it than greedy manufacturers maximizing their profit margins.
I think also that a competent frame builder has more options for handling, stiffness & comfort when designing a “compact”.