Everyone's body is different. As a perfect example, in your OP, you mentioned "not as hard as hiking". For me, hiking is much harder on my legs. Mostly because I don't hike as much as I ride, but I need more days off after a long hike than I do after a hard ride. Also, again, for me, it doesn't seem to matter what kinds of rides I had done previous days, some days I recover in a day, others I need two days, or more. I've done a hard ride on a Wednesday, and still not felt great until the next Sunday, whereas I might do an even harder ride on a Saturday, and still feel halfway decent the next day. Long story short, you'll have to learn to listen to your body, it will tell you.