If the easier rides are at different pedaling cadence that may explain it.
I get similar experience as you do when I'm doing lower HR ride at higher cadence. Shorter rides in my case but higher intensity. Zone 3 at high cadence will get my legs sore while zone 4 at significantly lower cadence will not get my legs sore at the same period of time.
Pedaling out of the saddle can also make a difference and we tend to do it in intervals at higher zones and can actually reduce soreness.