The part I'm not following is that one side is off 4mm and the other off 1mm. That means that the drops should be noticeably too narrow or too wide. And also, it will spring when you put a wheel in, so the difference will be averaged. So the alignment should be better with a wheel in. I would say that most bike mechanics really don't know much about this subject. In any event, the Park tool doesn't measure anything worthwhile as far as tire clearance goes. Or at least it would have to be off a lot more than 4mm to make much of a difference.
One other thing that could be off is the position of the chainstays on the bb. Some builders don't hold this too well. I would look at that before doing much grinding on the drops.