Alee Denham's quantitative study of this topic is fascinating to me, and really relevant to the kind of riding I do. This hard data takes a lot of the "weight weenie" out of me. He does make the disclaimer that these studies are about touring, primarily, not racing (where a few seconds can matter a lot).
The part of the study that looks at rolling resistance and aero effects of panniers also surprised me. I had these factors switched with weight before I saw his investigation using a power meter and stopwatch.