I always wear gloves just* for the protection (and without them some crashes would have been much more painful), but not all gloves are made equal, and it depends on the fit as well. If the glove fits with some play it tends to create comfort problems for me, unlike the ones which are just barely big enough to put them on, and provide a very snug fit. My guess is that if the fabric can move it can bunch up abd cause pressure.
I find that with cycling clothes in general, except shoes, going for the most snug fit is the most comfortable.
*Well, and wiping snot, obviously.