25 mph isn't all that fast if you are sucking someone's wheel and getting a draft. I'm not sure what your experience level is with pack riding but it's absolutely critical to get a draft off someone. Pulling in the front is a completely different topic of course but for now you should be focused on trying to hang near the front.
Regarding 5 vs. 4, sometimes Tri guys and strong dirt riders drop in on the 5 races and some of those guys are crazy strong. The 4 dudes are most likely jockeying for position for the primes and final win. That's the difference between racing and just out for some hammering.