Where in the group ride are you struggling? Is it when the group is going fast on the flats, downhills, or on climbs? Unless they're going well over 20 mph, my guess is that your bike should be fine and it's mostly body position/drag, fitness, and maybe group riding skills that you need to improve. In the winter, I ride my CX bike with a single 40t chainring and 11-36t cassette and keep up with the 18-20 mph group rides, so I don't think overhauling your gearing is crucial.
The most important factor is not equipment choice but fitness. Generally it's the climbing that new riders struggle with, and it's mostly a function of your power to weight ratio. You may be able to lose some weight from a gravel bike as the frame and especially wheels tend to be heavier for off-road use, but honestly that's like 4ish pounds you could save at the cost of thousands of dollars. Consider the percentage of total weight that 4 lbs would be.
Aerodynamic drag from your body will make up a big portion of the forces you must overcome at speed. Are you wear baggy clothes and is your bike set up in a very upright position? Those are both bad for speed. You might not be comfortable moving some spacers from under your stem or flipping it to get lower, but it's something you can experiment with. Tight-fitting clothes are just plain faster.
You can hang with others in a paceline if you stay close and save energy in the draft. It's not hard, just ride more with others to get used to it. Energy saved when you don't need it is invaluable for when the riding gets tougher.
If you're sure you want a pure road bike and are fine with ditching the versatility of your gravel bike then by all means look something lighter and more aerodynamic. Or you could compromise, drop 1-2 grand on a lighter yet more aero wheelset that's optimized for 25 or 28mm tires, and keep your gravel bike. You'll get a little bit faster in most conditions, but don't expect a miracle.