I used to wear gloves, not it does not matter on short rides.
During long distance cycling events gloves will rub and irritate skin in wrong places(tried many gloves in different sizes).
Riding without gloves feels more comfortable for a while (in my case I can ride longer without gloves).
Although I still carry gloves with me(sometimes a couple of pairs) and put them on for a few hours at a time, so the feel on the handlebar would be different. Also switching between wearing/not wearing gloves changes pressure points on my hands on long rides.