Are you racing?
Here's my take: I've always felt that my Aluminum Cannondale Synapse was too heavy. Last year I lost about 30 pounds, and have leveled off down 25. But it was the bike that was too heavy.
This year I got a commuter; an Aluminum Cannondale Quick CX. I put fenders on it, a Tubus rack, and panniers. My commute home is +630 feet gain over about 4.2 miles. If I commute all the way (avoiding the light rail) it's 1650 ft gain over 18.5 miles. I've put a thousand miles on the commuter bike this doing just those sorts of rides; 4.1 miles to the train station, 4.2 to get home, or maybe 18.5 to get home. I still jump onto my road bike, the Synapse for fitness riding and exploring.
When I ride home, I ride home with one or sometimes even two laptops in the panniers, a change of clothes, headphones, and so on. The commuter bike is anything but lightweight. Some of the inclines along the 18.5 mile commute home reach 17%. I've also taken that bike up Little Cottonwood Canyon, which is a 3500 foot climb in about ten miles.
I've learned that my Synapse isn't too heavy. Neither is my Quick CX commuter. I am measurably slower on the Quick CX, but I still get there within a couple of minutes of when I would have arrived on the road bike (wherever here is).
Weight matters if you're trying to beat a competitor who is able to put out more watts per kilogram of total weight than you. ...when that happens, to keep up you have to either shed weight or increase power. This is where a lighter bike makes all the difference. But my loaded commuter is still fun to ride; you get used to the panniers, and barely even notice they're there.
This isn't to discount the notion that reducing weight in the bike helps. It does. But you can still have fun on a heavier bike.