I agree with the others who think of your fork as the primary cause of the problem. It could also be because of misalignment of the main triangle. Trying to fix this problem by changing stem and/or handlebars is akin to performing a colonoscopy by threading the endoscope down the patient's throat