I worked on a farm 6 days a week for a couple years. I found that once my body got used to it I was able to do things on top of the physical job. I think however what is key is your body getting used to the labor first. I would start doing short rides on your days off at least as soon as possilbe and then work in some days you have work. Eventually you will adjust.