Yes, seems that the mismatch is the root of your issue. The proper ferrule should do the trick. Get the same size as the cable, usually 4mm these days, (sometimes 4.5mm) and it should be good. Steel or alloy is less troublesome for shifting adjustment, just check for oxidation occasionally.
If you get plastic ferrules, a method I use is to drop a small washer in the ferrule before pushing the cable in. 2mm I.D washer is probably what I have. Keeps the housing strand from pushing through the plastic and "settles-in" quicker into the stops. Less adjustments needed during/after the first few rides.