I'm a local and I ride to and over the Divide fairly often, on tours and day trips. I have a friend in Breckenridge and ride over to his house on the first day. So I don't have any lodging experience in the area. I think it's pretty expensive.
I strongly agree with the suggestion to reverse your direction. If you do that, I think your most difficult day would be between Keystone and Georgetown. If weather cooperates and you have a good morning to climb Loveland Pass, and you've slowly acclimated over several days, you stand a good chance.
You'd also want to stay in Vail to get in good position to climb Vail Pass on a nice morning.
I think someone has already mentioned the usual afternoon thunderstorms in the Rockies. The high passes are not a good place to be in the afternoon.
One of the harder climbs for me mentally is Floyd Hill, heading east out of Idaho Springs. It's a 1000' grind on the shoulder of US 40 at the end of the day. You may want to plan a night in Idaho Springs and tackle that one in the morning as well. The grade and elevation aren't hard.
The steepest grade is a short climb on the new MUP between Chief Hosa and Evergreen. That's easy to walk if you need to.