I don't think what you are describing is "joining" their group, you aren't riding close enough in for there to need to be any coordination. If you were going to ride close enough to draft, you would need permission, but following them at a distance is something you're free to do. It's normal for someone to be behind them and won't affect their ride in any way. Asking them is probably just going to confuse the issue because now they have to figure out if they have to worry about dropping you.
Humblebrag apologies, but I've had the opposite problem on some long rides. I ride pretty fast, and occasionally a group will pass me barely by pushing themselves really hard, and then the pass peters out because they run out of gas. Now I'm in the middle of the group, or the group is just ahead of me, but slowed down less than my cruising speed. I am not going to fall back in that situation because I don't want to lose my pace, and I figure if they didn't want me to draft, they shouldn't have passed me like that. Usually after about a half mile to a mile of drafting, I can shoot past them, and they're pretty spent.
If they're a good, coordinated group, btw, they'll probably lose you well before they know you're there. If not, following them as a speed test violates no etiquette rules I know of. Wheel sucking is an entirely different matter, however.