The first time I rode on the roads of somewhat to rather hilly north Raleigh, I thought, or maybe that should be, I hoped I would die. 10.5 or 11 mile of down and up, rinse repeat, were a LOT harder than my previous riding on the bikepaths. I kept riding. Nothing spectacular, but I've done alright for myself.
Aside from your possible health issues, and perhaps that should be, even including your health issues, there is only one metric that matters:
Are you having fun?
If the answer is yes, nothing else matters.
[Except the family, that is. They always come first, second, and there is no third.]