I did my first birthday ride at age 50. I've only missed three or four since then, usually for a good reason. Recovering from a heart attack was a good reason, as was this year's staying off the bike for 7 weeks on doctor's orders. Still hope to get a late BD ride in this year -- maybe adding a "year" in the process.
As Feynman titled his book, "Why do you care what other people think?" I enjoy the challenge, yes, and also being outside, riding my bike for a long ride, and the feeling of "See, I can still do it!" when I'm done.