That Doctor says some weird stuff.
I raised similar questions with four different Cardiologists in the past 2 years. I'll take their advice. Personally, I do a lot of aerobic exercise. I have zero blockages, good blood pressure, and am on no medicines. After reviewing everything, the Russian ElectroPhysiologist looks at me and says, "Total Respect, Total Respect"....."All my patients have problems created by lifestyle, I know how hard it must be to be fit like 20 year old....." When I was checking out, the receptionist asked when he wants to see me again. I said, "never"......she said, "impossible, that has never happened, let me ask him".
I find that recovery from hard days riding take me longer and sometimes need a recovery day more often.
You have to find the answer from a medical professional but I would get another opinion. Pickleball.