Panniers are less aerodynamic and will slow you down a bit.
The main advantage is ease of use. All those little bikepacking pouches are a pain in the ass to load and unload. It comes down to how much you need to carry. If you can fit everything into two bikepacking bags, fine. But I see people with 5-7 pouches strapped everywhere like mushrooms on their bikes. At that point it's just ridiculous. Get a pair of panniers and be done with it.