Cold slows me down the most. Cold air further aggravates my exercised-induced asthma and often makes me feel "wheezy" on harder efforts. My power numbers are typically down and my joints don't really like the cold. Anything below freezing and I'd rather just ride on the trainer. It doesn't help that I live in a mountainous area, so the extended-length descents are brutal in the cold, even when they aren't snowy/icy.
While it may not feel pleasant, extreme heat doesn't seem to have a detrimental effect on my power numbers. Some of my PRs on local climbs have come at temps near or above 100F mark and my best race results have typically been in hot temps.