If you get a bike with a one piece crank (common on cheaper small bikes), remove the crank as well, as they tend to stick out wide enough to hit little legs.
If you can get a s/h balance bike, tho, go with it. We nearly always go that route snce our club bought 3 back in about 2000. we teach about 40 kids from 3 upwards every year.
Training wheels are the INVENTION OF THE DEVIL! (Sorry about the shouting). They train nothing, except how not to balance and balancing is the heart of riding a bike. They should be banned under whatever is your equivalent of the UK's Trades Description Act as a misleadingly described product.