After years on a MTB based commuter I acquired an old road bike which I rode almost exclusively for a whole summer. Even now, with the new racked and fendered semi-touring commuter, I still like to ride the old roadie in to work. It has lights, air-horn, bell and saddle bag, plus a small frame bag. I've always used a back pack, which I rack on the other two bikes, but wear on the roadbike. What works for me is letting the straps out long and resting the bag on the small of my back, rather than tight against my spine. Even though I put a smaller chainring upfront, it's no climber, and I spend more time climbing out of the saddle than on my other bikes (practically no time on the MTB). I also went to wider tires, 700x32 vs the 23s it came with. My backpack is about 15-20lbs in the summer and 20-30lbs in the winter, but resting on the small of my back I don't feel it much as I use a take-a-look glasses-mount mirror and don't have to twist my body around often.
Aside from no fenders and downtube shifters I like this bike for its relative nimbleness, relative lightness and super comfortable super-flexing steel frame. And like you and many others I have found that although my three bikes settle in to three different crusing speeds, my commute times tend to be similar due to traffic, lights and conditions.
And even though the downtube shifters are less convenient, require a reach down to shift, I find that keeps me from fidgiting and constantly shifting. And I really like the direct feel of friction shifters, at least on my bike.