I mean I know plenty of people who ride thousands of miles a year who are slow. Literally every training program I've ever seen is basically variations on high intensity intervals. I didn't start getting faster until I did some targeted training.
Riding with other fast people also helps.