The question is basically unanswerable without more info--"does it matter" and "major downer" are relative terms. It really depends what you're using the bike for, but on the whole, there's many factors that will affect the speed of the bike more than a couple pounds of bike weight.