I see no reason why you shouldn't pass if you are traveling faster at that time. Though I sometimes ask myself if I'm going to be maintaining the particular effort I'm doing for very long after the intended pass. I don't ride the same effort for my entire route and some times I'm slower and resting and other times faster and burning up the useable energy.
So it might be that someone I pass might have to pass me a few hills later. Doesn't bother me. I just assume the parts of the ride where they ride hard don't match up with mine. Is it a big deal to some?