That you haven't all 4 ridden together may cause you to have a 3-person paceline somewhere around mile 75. Even though it's not competitive, it does help to have everyone on the same page.
If wind isn't a factor, rotating out left or right shouldn't matter. Unless the dude's Zoolander or something.
