I use 165's too although basically all my distance riding is on a fixie, too. I'm 5'6". At a certain point years ago, around when I started doing a much higher percentage of my miles fixed, I realized that I felt like I was pedaling squares on the geared bikes (with 170's), on the rare occasions that I rode them. I switched everything over to 165's and it felt much more natural. I can't be sure that it isn't just what I'm used to, but I definitely feel more comfortable overall with 165s.
It's a complex picture, though. In addition to all the aforementioned stuff about angular velocity and linear velocity, there's the angle of your knee at the top of the pedal stroke, the amount of leg extension at the bottom, and the gears you have available. Longer cranks give you more leverage, but so do lower gears. See Sheldon Brown's discussion of what he calls "gain ratio" for more on that.
Generally, it's probably fair to say that riders with longer legs tend to prefer longer cranks than riders with shorter legs, but there are many other factors that can come into play. One of my idle theories is that riders with bulkier/heavier legs are more likely to prefer shorter cranks because the mass of the leg doesn't have to go up and down as far, but maybe that's just the result of descending so many long hills on a fixie (having legs on the bulkier side). I do think riding style and body type are important factors, and if you feel better with one crank length than another, that is more convincing to me than what conventional wisdom would dictate.