When we had a bike club in this town and I was an active member, I did a lot of group rides on one or another of my fixed-gear bikes. My memory is that I hung with them pretty well, but I was in shape then, too, and riding 2,00-3,000 miles annually.
In terms of my speed, for general riding around this area's rolling terrain there isn't really a significant difference between whether I ride fixed or multi-speed. I can have a lovely and enjoyable and even challenging ride either way, whether solo or with others who are seeking a comparable ride experience, and I bet you would find the same is true for you.
In terms of riding with others, if the goal is to go fast and hammer, no, you will want to ride something comparable to the pack. I've been there and done that and could happily spend the rest of my life far from modern roadie culture and its conspicuous consumption - but that's a big part of the appeal of a fixed-gear for me, it's an upraised middle finger to the constant consumption/obsolescence/replacement cycle.