I've done the same thing. Eventually I discovered the love of extreme bar drop. Now Iove having my bars 170mm below the saddle. I would have never believed this could be comfortable until I tried it with what I then considered a too-small frame. I had this small framed bike and just experimented with increasing bar drop after I saw professional cyclists who ride all day every day in this radically aero position. I shocked to discover how much I love it. It's counter-intuitive but lowering the bars relieves pressure on my arms and hands. Previously I was having discomfort from shoulder, neck, and back pain, hand and arm numbness.
I believe that bicycle fit is often a psychological hurdle. People just have erroneous beliefs about what positions can be comfortable, and that riding upright is somehow better than riding aero. I'm sure my setup looks torturous to other people, but to me it feels terrific.
Even my mountain bike has to be set up this way. I've quit tweaking my riding position after doing this.
