Phone is always on but I don't go out of my way if I hear an alert. I check when I take a break. I too use it for riding apps. It's always in one of my jersey pockets.
How I use it while riding is how I use it when driving, I NEVER get distracted unlike those fools out there.