Every time I get a significantly lighter bike, I can feel the acceleration surges as I sprint from a stop or increase my speed rapidly. Also, some of the hills that use to give me issues become no longer an issue.
As for increasing my average speed, not so much. Maybe a 1/2 to 1 mph. Lighter does let me ride farther and retain more energy.
If you live in constantly rolling terrain as I do, I'd get as light a bike as you can reasonably afford if distance and climbing are something you want to improve your numbers for.