I agree. Polarized lenses make it harder to see in some light conditions, and they black out the screen on my phone/garmin/general electronics. I have a pilot friend who has to buy non-polarized for reading instruments and determining cloud layers from above etc, and he has a similar problem to you; can’t get good glasses with non-polarized lenses without spending a fortune on special order shades.
Someone will probably be along in short order to tell us that true polarization that you get with $500 Oakleys shows you the true colours of the world, and without it you’re going to crash into a tree. Don’t listen to them.