Update:
So I've now figured out that it is 100% the cog moving on the threads. If I stomp on the pedal backwards, it will make the noise, and then if I do the same thing forwards it will do it again. So the cog is moving a tiny bit on the hub (about the same amount as what you feel with a loose chain on a fixed bike if that makes sense).
I'm going to take everything apart and re tighten with anti seize (I used Phil grease before).
What are the chances that this has damaged the hub at all? It's had two rides with maybe 15 miles total, not to mention hopping around my yard trying to figure out what was making the noise.
Thanks.