There is a petite woman in our bike club who is a long time cyclist and has no trouble keeping up with the fast group so it is mostly a matter of cycling fitness. At the same time, equipment does make a difference. I suspect your bike has fatter tires than your husband's bike. Narrower tires of, say 25 or 28 mm are lighter so can spin faster, maybe at slightly less effort and at your weight, with less air pressure than your husband would need, would provide a comfortable ride. Your current speed is a fine effort that can only get better.
Finally, many couples, as suggested in the an earlier post, ride a bicycle built for two, (a tandem), so they can remain together.