You're not taking advantage at all if the group insist you stay. I think you need to accept their version of events and be grateful, rather than being hard on yourself, or quitting because it's making YOU feel bad. You'll soon catch up - and they know it... so just relax and accept that, having presumably been there themselves, they probably know better than you. One day you'll be part of the main group, waiting for the next new member to catch up.