If you are used to sitting upright, then trying to change that might be more an effort of getting used to another position than it is anything else.
As I said, I'm not too flexible but getting aggressive with my position isn't an issue. If you have no need to get aero, then maybe it won't be for you. Don't be looking at aggressive fit bikes. Nothing looks worse aesthetically than a low stack bike with umpteen dozen spacers under the stem and a ridiculously angled stem to give more height to the bars.
But if you give up the desire to get more aero, you'll also give up a way to save energy on long rides. For a weak legged cyclist that I am, being aero helps me to have energy to finish a century reasonably strong. At least strong for me, but maybe not compared to the person that passed me in the last 100 meters.