“Are there any drawbacks or things to take into consideration other than the obvious pain of swapping the wheels every time I want to ride the other bike?”
nope. It’s just a pain.
I’ve done it, but that was back when I was still figuring fit out and on a college kid budget.
unless you’re the sort who meticulously cleans their entire bike after every ride as part of the whole ride activity, buy a big bag of Uline Nitrile gloves for the swaps.