I used to race CHUMP level xc(cat4 or cat3) and we were all up in the business of other riders if it was single track. Vividly remember dodging someone who fell/OTB and a bunch of us re clipping in. Lots of instances of riders having to slow down on descents due to a less "skilled" descender being in front. Same could happen on technical ascents, it was a domino effect where one guy messed up and the next had to deal or mess up as well.
A couple times racing at prairie city it was wet and we got the "gorilla snot"/"peanut butter" mud. Lots of slipping and sliding and not going down to frequently was a big deal.
I always got beat via people who either had great engines, or decent engines and great skill. I had the mass to be the fastest descender but I tended to run out of skill before my friend who rode motocross+dirt jumps throughout his childhood.
No one I saw wore armor since knee and elbow armor rubs bad and heats you up quicker. I know on normal rides I used knee armor when it was cool, on the hot days I valued my mental clarity more then the protection.
Yeah, I would say some other sports require MORE skill then road or mountain biking. To me that's okay. It's literally part of the game.