Handling and anything most people call "twitchy" are usually something you can easily get use to and it'll no longer seem twitchy. Between a 54 and a 56 probably not much difference at all for handling. Between a 54 and a 60cm, then probably noticeable. The 60cm being the ride of a luxury sedan vs the ride of a sports car for the 54. Whether you can fit on a 60 is a different matter altogether. However I'm only 5' 11" and I use to ride a 64 cm (25") bike comfortably for many years.
Your 54 cm Roubaix will have a lower stack, but still not as low as my 56 cm Tarmac. Whether that will directly translate into a more aggressive position might depend on how you hold your arms, very bent elbows or almost straight.
Instead of wondering, you really ought to just try them out. Even if they don't have the exact bike you want to ride, a 54 cm Roubaix and 56cm Roubaix in any tier level available will be pretty much the same fit. Though maybe the S-Works version might be newer frame design, but I haven't seen if they've changed that on the Roubaix if they even make an S-works model anymore.