I thought about this more last night. BTS, I think you are pulling up to events - races or hard group rides - seeing a lot of people in matching kits and expecting them to start working together to your disadvantage. I think you are psyching yourself out before you even start.
At your level, there is very little, if any, teamwork. And what teamwork there is, often isn't effective at determining the outcome of the race. Don't worry about what kit people are wearing.