Anything in the fit that doesn't properly set you on the saddle will cause problems. One sign of a good all-around fit besides comfort is that you will naturally and comfortably be balanced on the bike and therefore tend to be "anchored" to the one and same spot on the saddle.
As far as saddle height being off, modestly too high or even a little too high is worse than being correspondingly too low.