If I'd started to get comments like "Yer skivvies're showin'!" then I might take action.
But with the colors of things, or how things are matching (or, more likely, mismatching) ... meh. Is what it is. And if it's something I like then another's concern isn't really an issue for me. But, that's me.
I'd think if you're okay with the color of something you should be okay with it. Then, revel in the fact you're "a unique" out there. Isn't much of a bad thing.