When I do a long hard ride such that my legs hurt afterwards, I find that I'm in a bit of pain the following day, then often worse the second day and then substantially improved by the third day. This obviously means that muscle recovery from a hard ride takes at least two days for my 65 year old body. I'm guessing that you are interrupting your recovery periods with more muscle-damaging stress thereby re-injuring yourself and never completely recovering. This type of thing is a major reason that many athletes do cross-training with sports that use different muscle groups than their primary source. This isn't rocket science...if it hurts when you do something, stop doing that and try something else.