Yeah, the solution can have a number of forms. He can slow down. If the point of riding together is to enjoy each other's company and/or encourage joint exercise, him leaving you in the dust isn't doing either of you any good. The other side of that is you getting an e-bike. It's going to take significant effort for a 5'1" rider (who is presumably not as cycling fit) to maintain pace with a 6'4" rider (who is presumably pretty fit on a bicycle). You can keep up or pass him if you're on an e-bike and you both can then enjoy being out doors on bikes together.
This is exactly what my parents did. My dad bought my mom an e-bike for endurance, not speed (so she could ride more than 5 miles at a time). He liked it so much, he bought one for himself, too. And he converted his Trek DS. So they have three e-bikes, and they go on multi-day cycling trips now. E-bikes have been completely transformational for them, and they're a TON of fun, too!