I gave up riding for 25 years and returned in 2014 for similar reasons you listed. It will always be a risk but life is full of risks and I just try to mitigate as many as possible related to cycling. Due to arthritis, I can no longer hike more than two miles, run, or play basketball like I once did. I cannot tolerate the impacts. So I can either be on a rower, a trainer or find relatively safe cycling routes.
One regret I have in life is giving up cycling during my prime physically fit years. I missed out on life because of fear.