You're doing it right. It takes time to get the speed up, years really. Just keep at it. To speed the process, study up on training, read books, work up a year-long training plan, have a goal ride that you're training for, etc. 17-18 is a fast pace for recreational cyclists. The more miles you can do, the faster you'll get. Distance=strength. Gradually increase the length of your weekend ride and add a little to the mid-week rides, too. When you've worked up to riding 60-80 miles in the hills every weekend, you'll definitely notice a difference.
If you're under say 55, you can do intervals on one of your week-day rides as well as going all-out on the weekend ride. Some people only do intervals during the week and then use the weekend for a long moderate endurance ride, but that precludes the fun of joining a group ride. I prefer a hard weekend ride and then endurance work during the week.