Going with a shorter stem isn't going to compromise handling. The difference between two frame sizes is generally one stem size - so a 100mm stem on the larger size and a 110mm stem on the smaller one. That's all.
Larger modern bikes aren't more flexy. I don't know where people get that idea. It isn't really true of old steel bikes, either.