Very interesting. I get hands-off shimmy, but the bike is pretty darn stable at speed. I usually use evenly distributed panniers on the front, though.
On my recent ride there was an 11Km steep descent with hairpin turns that cars become stuck on. I was faster on the Birdy than I was on a hardtail mountain bike on the descent. I've never really loaded up though.
I also had chain derailment issues. That happened after the suitcase was smashed last year. I bent the front chainring badly enough that it derailed even with the chainguard on it. Putting a new one on straightened that out.
Running it with dual chainrings invites it to derail unless you can fit an outer chainguard and an inner damper such as the kind Dahon uses. (The Dahon one fits, but the way.) You may also want to check your chainline if you are not using your original bottom bracket. Finally, you might want to double check that the crank is on good and tight.