Good question! This is something I'm a little curious about too, though I don't think about it too much (I generally aim for distance and time riding rather than worrying about speed anywhere). I've also noticed I have a tendency to catch up/pass on the climbs and get passed on the flats. I generally do alright on the downhills but most of them around here curve near the base and I think most of my "gain" there is that I can take corners/curves faster on my randonneur with 42mm tires than most of the super light roadies can on their skinny guys.
So far the best thing I've found for not pushing myself too hard is to ride with other people who are also into casual riding. I tend to ride MUCH harder when I'm out on my own and while I'll still happily rack up 75+ miles I'm often beat at the end where I can spend a whole day out riding with other folks and feel up for a second lap when I get home.