Years ago, I had noticed on my panniers that after or during rides, one of the hooks would always bounce off the rack.
So one year I splurged and got an Arkel with cams that grip onto the back rack. Even then, I would secure it more with a carabiner.