Was it Greg Lemond or Eddy Merckx who said, "it doesn't get easier, you just go faster"
Re-reading some of OP's post, his/her expectations are a little unrealistic WRT times and speed.
I can't even reach a guaranteed 30kph on 100km distance on a not-so-hilly route, something that should be easily achievable on a roadbike by a fit person.
My training ride standard for 100km with 50 feet of climbing per mile (1% average for the distance)? 4 hours. It took me 4:23 yesterday with 66 feet per mile and 182 watts NP, a solid zone 2 ride. Of course, everyone's roads are different but doing 100km in 3 hours on randonneuring roads around where I live? I bet fewer than one out of a hundred randonneurs could do it and I am not one of them. OP's best bet is to adjust expectations.
It takes a lot of time, patience, and consistent training to improve one's aerobic system; otherwise, one becomes a patient. Avoiding sickness, injury, and burnout is very important. It is better to be the turtle than the hare. I am not sure how many times per year one can ride really hard one a brevet but for me, it is a couple per year.