Sounds like a lack of fitness to me, unless you have short arms to go with your short torso. I'm 168cm tall with an 83cm cycling inseam and 72-73cm saddle height. My wingspan is about 5cm greater than my height. My preferred stack is 525mm, no spacers and a -17 stem to produce a 10cm saddle to bar drop. If you're trying to produce a small 2-4cm saddle to bar drop, that's your real problem. Right now, I'm riding the smallest frame I've ever owned with a stack of 509mm. I have 30mm headset top cover instead of the standard 15mm headset top cover and a 15mm spacer. I still use a -17 x 110mm stem, with 80mm reach bars. If you have 100mm reach bars, I'd change those out.
Check your saddle to bar drop first, then report back.