You may not have the BMI of the top pro tour riders, but I wouldn't be overly concerned about your weight unless you have a goal to be a top hill climber. Lose 10 lbs over time?
Mix it up some. So some "routine" 20 mile rides. And, some 40+ mile rides knowing you'll be beat afterwards. Afternoons and evenings?
You could likely push it even longer... 50 miles? 100 miles?
Being tired after a hard workout may actually be a good thing.