There is a biological difference. Testosterone levels during development drive most of it (lean muscle mass, more bone density to support the muscles, less body fat to push up hills, etc).
I briefly went through last year's time trial results from multiple races. TT's were chosen to remove the "lack of aggression" and small field size factors that I see in women's racing. Every single race result showed the same pattern: The winning time of the P/1/2 women wouldn't get a podium spot in the men's 3 category and usually wouldn't crack the top 60% of the men's P/1/2 field. Same course, same conditions, same gear.
And also, inb4 TL.