There's a sticky at the top of this subforum about this. I'd suggest taking a little 15min read through that topic, it covers a lot of what you're asking or interested in.
Otherwise, 5's (and for that matter non racers that jump into weeknight worlds) chase EVERYTHING that moves. There's zero recognition that ain't nobody getting away alone. So, they see someone make a move or little attack and they pounce. That drives up the avg speed. So a 3/4 race can often be a slower avg speed.
If there are plenty to do, just do more of them. If you're of age, double up the "open" and "masters" race for your category. If there aren't join a weeknight worlds ride to do weekly.