In my size at least, Endurace isn't an upright, slack frame compared to an Emonda, Dogma F or the SuperSix, all of which are race bikes. Bikeinsights.com can be used to compare different frames.
Rarely does a pound or two off a bike result in a faster ride compared to a more aero rider and more aero bike. The exception is a pure hillclimb. The other exception is in a race where one's W/Kg is marginal and a couple of pounds could decide whether one is dropped or hangs with the pack. Riding in S Cal, I'd do with aero first all day long. There are ways to calculate using bestbikesplits or using Gribble. Comfort is probably most important because if you cannot hold the proper position, you will be sitting bolt upright and that is the slowest of all.