I think it matters enough to factor it in when you're looking to buy a new bike, but not so much that you should stop riding a bike you're already happy with.
I will say that I am on an aero bike and love it, and still dream about an Allez Sprint. Such a nice frame...