I hate the stop sign scenario and I usually do the same as you, not looking at the car.
Sometimes there is too much glare to see through their windshield anyway.
But if they do obviously wave me on, I'll take it even if they had the right of way. Morons.