Personally from many years of mountain biking the rule I always followed is that on any trail (especially trails where hikers and potentially horses) that is used for both directions uphill has right of way. There is no way a bike going downhill on such a trail should be going faster than visibility ahead allows and it is much harder for an uphill rider to get going again. There are trails that are designed to be downhill only as well as some that are generally considered to be downhill trails where it is accepted that downhill has right of way.
I try to always give uphill the right of way unless they clearly stop first and wave me on.