Chain tension for single speed bikes is a big misnomer. The correct "tension" is actually a very slight amount of slack. If the chain is actually adjusted for actual tension than it'll do what you're getting. And it's really bad for the chain, all the gearing, the freewheel bearings and the bottom bracket all at the same time.
As you back pedal the cranks the chain will alter the chain tension due to a variety of manufacturing differences and slight out of roundness issues in all the various components. What you want to avoid is a spot anywhere in about 15 to 20 crank rotations where the tension actually goes tight. Very little slack to where it's not felt is fine. But if the crank seems to feel any resistance as if going over a "bump" and kicking forward as it passes then you are getting a tight spot and need to slack things off a hair. Meanwhile at the most loose spot you should only be able to move the chain at mid run up and down perhaps in an extreme case of 1 inch. Ideally the chain slack would remain constant at around 1/2" or perhaps slightly less. But in the real world I'd be OK with from not quite tight at the tightest to up to around 3/4" of slack at the most loose.
If I find I'm getting an inch of slack or more I'll try to see if I can move the chain ring around on the crank spider or I'll change the crank set to a different quadrant on the square taper or try some other tricks to try to get the chain ring running more concentric.... hopefully.... If the issue is at the freewheel cog then not much can be done other than to get a different freewheel and hope it is made more concentrically.
And Clifford also makes a great point about the quick release issue tightening the cones and causing drag. So much so that I made up a couple of thick plate washers that pass a QR skewer and let's me tension the QR. Just heavy 1/8 steel squares about 3/4 to 1" square and 7/32 (a touch bigger than the 5mm needed) in the middle. This leaves the cone and locking nuts accessible to adjust the preload with QR tension in place.
After you carefully adjust for a nice running bearing and release the QR tension you'll be surprised at the amount of play in the cones.
Hope that helps out.