I spoke to my GP, and he said to stop taking the BP meds, keep track of my blood pressure, and call him in 2 weeks. If it goes back up, he'll put me on something different and see if it works better.
Upside, I stopped taking the norvasc Tuesday, and my energy level is coming back! Went for a ride yesterday, same basic route, did better on all segments! My ankles weren't swollen after, either.
However, my blood pressure is going back up, so I suspect I will have to be on something for it. We'll see.
Thanks for all the replies, advice, and moral support, guys!