Another vote for caliper spacers. I've had similar issues with bikes I've owned and a good solid spacer has worked without any issues. Your local bike shop should have some in stock with the correct tolerances as opposed to getting a sloppy fitting one at the hardware store or stacking washers.