I tend to think the main limiter with amateurs is the amount of time and focus they are willing and/or able to dedicate to the sport. It takes so, so much work to hit your genetic potential, and the amount of time it takes to get there is so unreasonable that most people don't do it. Pursuing it with all you've got it probably takes like a decade plus to get there. It's seriously stupid to do it unless people give you money to ride a bike, and even then you're not exactly raking it in.
If we don't factor in anything besides genetic potential, I would be amazed if most people aren't capable of much more than they think they are. If we factor in that real life is a thing and making money is a necessity and drinking beer is fun and training when it's 20 degrees out sucks, then I guess there are plenty of people right at their limit.