I've only ridden handlebar bag with rear load and LowRider panniers and varying loads in back. The former until about 1980 when I put LowRiders on my Peter Mooney. That was a revelation and I've never even considered doing anything else. I very quickly figured out the heavy stuff needed to be in the front panniers and if there was nothing else, leave the rears at home.
But - heavily loaded LowRider panniers 1) put a big load on the front rim when you hit say a pothole edge. Blown and damaged tires and dented rims happen unless tires are big enough with enough air and everything is strong enough. 2) I haven't studied this systematically but my experience suggests that flexy forks especially forks with a lot of rake do poorly with the heavy panniers. I farmers market with one of those skinny forked Raleigh Competitions. Loaded it is a flexible mess. Haven't noticed speed issues but accelerating while turning hard (say to head up a steep hill) is interesting! By contrast, my much stiffer Mooney with it's much straighter and stiffer fork (and about 2 degrees steeper head tube) behaves beautifully with any amount of weight in those panniers.
Other comments re: handling - in marginal road conditions, loaded LowRiders are wonderful. Angled railroad tracks and snow; a huge help. For my commute to campus in Ann Arbor from nearly the highest point in town. I used to carry extra books just for the steering stability. Allowed my to push my weight back over the rear tire when otherwise I'd be weaving around and skidding and ultimately walking. RR tracks - loaded LowRiders = lower heart rate spikes when you cross the angled ones.
Just my observations after about 45,000 miles with LowRiders