You’ll find a lot of folks trying to get away with doing things that are contrary to manufacturer recommendations. Doesn’t make it a good idea.
Specialized also states in their carbon steerer instructions that spacers are not to be placed above the stem except for initial fitting. For permanent installation the steerer tube is to be cut to the correct length. The reason has to go with how the expansion plug is installed. Doing otherwise can lead to failure and voids the warranty. Basically there is no going back.
What I have done to drop the stem without cutting the steerer tube is to install the stem with a reverse angle. Reverse angled stems are perfectly sound and more common than you might think.
Good luck!