If you want them to match, get a matching color. If it makes no difference to you, don't worry about what others might be thinking.
I personally try to match my clothes when riding, even on the Mtb but I don't go out of my way to do so or sweat it when I don't.
Wear what makes you happy without breaking the bank