I am rather good overall physical shape. I started off with different entry level bikes before scoring a good deal on a carbon mountain bike.
Even with my fair amount of skill and physical conditioning, I am well within the limits of my Felt Doctrine.
Witu that being said, yes, a better bike is totally worth it and does greatly improve your experience if you are confident in your abilities.