It is not imaginary, but some people are more sensitive than others to small changes. For example, Mark Cavendish can deal with very big variations in saddle height, he changes it by centimeters at a time depending on what he feels is appropriate to the terrain.
I am more in your camp, small differences matter to me. For example, I will tend to get knee pain in my right knee if the cleat on that shoe is misaligned. Despite the fact that I ride with float, I have to have the cleat adjusted so that my toe is turned in fractionally. A difference of only about 5 - 10 degrees is crucial.
Saddle tilt is another. It has to be fractionally nose-up, for me, but I mean fractionally, barely visible to the eye. If it is level I will find it uncomfortable.
As for saddle height, 5mm is enough for me to notice the difference.