I regularly ride with fixed gear riders and it isn't a problem. In fact on the flats, they can sprint out the gate faster than any of the geared cyclists. I especially make it a point to draft the bigger fixed gear riders because they're great for blocking the wind and can pull better on shorter road segments. Though when we hit the hills, I unfortunately can't reciprocate the favor and leave them behind!
