IME, the front lowrider / pannier setup has the least negative impact on handling and in fact improves stability. The rule of thumb seems to be the closer the weight is to the steering axis, the better, and front panniers put the weight right there.
SP
Bend, OR