I read a scientific report a year or two ago about energy usage during aerobic activity. Specifically, they tested athletes on a cycling ergometer. The athletes rode at a set % of either max heart rate or FTP (I can't remember which) and then took tissue samples.
At rest almost all energy usage is supplied by free fatty acids and blood glucose. As aerobic output increased energy from the aforementioned sources stayed the same but the new energy sources were glycogen and fat from the muscles being used. Fat and glycogen usage increased until about 65% capacity. After that, fat usage tanked and glycogen usage soared.
In other words, if you are riding with fitter people it's probable you are using vastly more glycogen than they are so you will have to refuel while riding and they might not. Also, it's possible that your lactate levels increased to the point where you have to power down. The others are fitter and, therefore, better able to cycle the lactate they are producing. One is a fueling issue, the other is a fitness issue.
The glycogen issue is something that can be fixed quite easily. Eat while you ride. And don't wait until you are hungry. There's only so much carbohydrates you can process in an hour. A good estimate is 1g/min. The other is a fitness issue that just takes riding more and training better. Try some high-intensity intervals, for example.