As I'm sure many have already said, weight makes a difference proportional to your total weight (you and the bike) for climbs and to a very small extent on the flats since rolling resistance is proportional to weight. So, 2-3 lbs difference may only be <2% total weight. But weight does change the feel of the bike and riders are not machines and I believe feel matters.
The bike matters, but much more so once you plateau in your physical gains. And you will plateau and before that the gains will come very very slowly.
There is one place I think the bike really matters, it is the tires. If nothing else, get good tires like Continental GP 5000 and run tubeless if possible. The difference over even medium performance tires is not marginal. If you're already nearing the best you can do physically, it could represent the equivalent of months if not years of serious training. Does that matter? Perhaps not if you aren't competing. Even if you don't compete, but just want to go faster for fun, then it matters.
Beyond that, better bikes tend to be more comfortable. At least effort has been put into designing in comfort. A more comfortable bike tends to be a faster bike because you fatigue less during a ride. Be sure to get a pro bike fit to take advantage of this.
I have what by most anybody's standard is a very expensive bike, and I'm more than happy I spent the money. It's very much a law of diminishing returns as you spend more, but it puts a smile on my face just looking at it, and more so riding it.