It varies considerably between individuals and isn't well-correlated with fitness. For health considerations there usually aren't any significant issues below 8000' which is the maximum pressure-altitude mandated for commercial airline flights, but, as noted by Robert, many people notice effects when doing strenuous exercise at slightly lower altitudes. But I wouldn't expect you to feel much of an effect on a ride that stays below 4000'.