While the pure aerodynamic difference between an aero road frame and a "standard" road frame is going to amount to only around 1% plus change (which isn't big, but given there are hardly any real downsides to the aero frame except a small bit of weight, isn't something to scoff at either) with the same rider position, it might be impossible to get the same position on the "endurance" geometry frame with a much taller headtube and larger stack height, and that increases the difference somewhat. For me, going about 2.5cm-2.7cm lower (removed a spacer and went to a -17 stem from a -6 one) and from a 40cm bar to a 36cm bar was worth about 1.5-2% on a typical flat-ish course at the same power, and after a short while of getting used to it there was no downside to the reduced height of the bar (sometimes I fancy that slightly more width might be of small benefit when climbing, but I'm not sure).
If you're the kind of person who would worry about "well, what if I had a faster bike", then just get a race bike outright.
Also, I find the aero bikes with straight-ish top tubes to look, well, cool - the agressive sloping of some compact frame bikes just doesn't do it for me visually.