I have a Tiagra equipped aluminum road bike and an Ultegra Di2 equipped carbon road bike (Roubaix). Fit is basically the same. I'm fractions of a mph faster on the more expensive bike, while I'm 1+ mph faster on the aluminum bike when I use the clip on aero bars. The takeaway is that an expensive endurance bike won't make you nearly as fast as being able to put out power in a more aero position. Sure, the more expensive bike is lighter and climbs and descends better, but depending on your weight, the difference may be neglible.
If you want to get significantly faster, a better use for your money is to buy a power meter and/or indoor trainer and follow a training plan from TrainerRoad, the Sufferfest, or Zwift. And for free, start getting used to a more aero riding position.
If a nicer bike will make you want to ride more, then by all means treat yourself. I did, and I have no regrets.