the fork geometry wasnt designed to have a load on it. look up "low trail fork" and youll find ten thousand randonneur threads and articles about what its all about
with a bag on my trek the front end is squirrely as hell, but on my bridgestone its perfect and i can ride for miles no hands