Actually, it isn't speculation. Armstrong did, in fact, test positive for EPO in retroactive testing conducted in 2005. There were 6 samples that tested positive for EPO from his 1999 Tour win. Just because he was not sanctioned does not mean that he didn't test positive. He did. This coupled with other strong evidence linking him to a doctor convicted of providing EPO to several cyclists makes Armstrong's Tour de France wins as not legitimate.
It's like saying that Barry Bonds is the greatest ever baseball player. The only evidence pointing to Barry's drug usage is his link to a trainer who was busted for providing drugs to other players. The evidence against Armstrong is far greater.