I'm concerned about a few things.
#1 it's not about what goes on during the 2-4 hours on the bike. What are you doing to help yourself during the 20-22 hours OFF the bike
#2 you've got to learn how to go slow before you learn how to go fast
I am almost 50. I am an A-/B+ rider, play hockey and work 50 hours a week. Wife, 3 kids, etc. I've got to be efficient with my training, time.
Monday, walk the bike
Tuesday, tempo with intervals
Wednesday, tempo
Thursday, fast, with intervals
Friday, tourist. walk the bike
Saturday and Sunday, fast A group rides.
Is there any better cross-training than hockey? cross biking?
You know your body better than anyone else. Train as you feel.