As long as you aren't a Pro rider it's not an issue. Many orgs have rules on socks. UCI just changed theirs up.
I can't see riding without socks myself. I need to have a little bit of compression on my feet to keep them from swelling a tad at mile 35-40 and making my feet ache for the next 10 miles till it goes away or I become numb to the sensation.
But you are welcome to do what works for you.
On the other subject brought up ... I think my bright yellow socks might be faster.... but the red socks I wear too aren't much slower!