It's possible to get a good fit with the right stem and bar choices. I actually sized up for my track frame (Dixie Flyer 58) compared to my road frame (56). But by using a -27 degree 85mm stem for the aerobars I can match my road TT fit (which is pretty agressive). For drop bars I use a +6 degree 110mm stem with shallow drop bars (Pro Vibe 7S). The drop bar setup is longer/lower than my road fit, but the more agressive position works well for on the track since I don't have to worry about being comfortable for 3-4 hour rides.
This approach wouldn't work with a lot of frames, I found all the cheaper frames the head tubes are way too tall for me to get a good fit for both. The BTB runs long/low though, so it worked out. The higher-end Felt frame also would have worked, but was more than I was willing to spend. Dolan DF4 also might have worked, but none of the sub-$1K frames I could fine had the right geometry.
The problem is not simply getting the reach shorter for aerobars or longer for drop bars. The problem is that when you change the reach (for whatever bars) you change the handling of the bike because you are not changing the top tube length. Therefore you are simply adjusting where your bars are in relation to the front axle.
It's kind of hard to explain.
Take a photo of a bike then photoshop erase everything except the rider and everything from the headset forward (bars, stem, fork, front wheel). There is a sweet spot for the handlebars/aerobars above the front axle. The contact points on the bars, front axle of the wheel, and headset make a triangle. If you shorten/lengthen the stem, you are affecting that triangle.
Here are some pics from the internet to express what I'm trying to say. Imagine if these frames had the same geometry and TT length. Note where the stem/bar combos put the hands.
Remember, materials don't affect steering handling. Changing the geometry affects the steering handling. Changing the stem angle/length changes the geometry of the bike.
These bikes will handle differently.
This is why bikes handle awkwardly when people are first trying aerobars and use the same stem that they use with drop bars and they have a hard time handling the bike. Then they put on a much shorter stem and things improve tremendously...but a different frame with that shorter stem could be even better.
The Mass Start / Sprint riding position is very different than the Pursuit riding position.