How much left to right can you turn the fork? Seems like you'd be able watch the center of the cap screw for the stem and note if it orbits instead of staying dead center when you turn the fork to it's extremes.
Might help to have the bike frame clamped to something to keep it still.