This is a nifty little tool:
Bike Calculator
Based on my amateurish understanding of physics...everything it calculates seems reasonably accurate.
Long story short if you want to avoid a few days of research and/or going back and forth on the forums: effect on speed both on flats and hills is measurable. It's probably measurable, but functionally negligible on flats unless you are literally a pro racer, assuming we're talking a few pounds and not 50+. On hills it makes a bigger difference. Will 5 lbs make a big difference on a hill? The bike calculator says 5lbs will make a difference of ~0.2mph on a 5% grade, which sounds about right to me. The calculator says the change in speed is about half that on flats.
As a test: leave your 2 water bottles, keys and phone at home for a ride. That's ~5lbs.