Yep, there's a difference. Mostly because biology rigs the game.
There's no substitute for a lifetime of inhabiting a body flooded with testosterone during formative years, not to mention the general differences between how boys and girls are psychologically conditioned. Fortunately the latter is less a factor now and more women have a better chance at maximizing physical potential without the baggage of inhibiting psychological and cultural conditioning.
A local woman pro who's in her 30s has many of the QOMs in our area. I'm a 62 y/o man, only middling fast even in my age group, and have faster times than hers on some Strava segments. But she's a better climber and has better stamina. To equal her over distance I need to have an exceptionally good day.
But testosterone rapidly diminishes with age and cannot be easily replenished without other complications. In my case, it might offer some short term improvements in energy, recuperation and strength, but due to an auto-immune disorder, a dysfunctional thyroid and early onset osteopenia, I'd risk long term complications that might not be worthwhile.
If I were a good candidate for medically supervised testosterone therapy, HGH, etc., I'd easily beat the best times of every local woman pro.
And I'd still be slower than some of the fastest guys my age.
And I still wouldn't be able to keep up with Emma Pooley on mountain climbs.
Biology is unfair. Even when women athletes have been caught using PEDs it doesn't magically boost them to levels comparable to men. It barely gives them an edge over their (presumably clean) female competitors.
And there are exceptions. Even with my background in amateur boxing, I still wouldn't step into the ring with some women pro boxers my size, including Layla Ali. Even if I could play tennis Serena and Venus Williams would still kick my butt.
And there are differences in physiology, metabolism and balance that may favor some women in endurance events, gymnastics, etc.