I've been a little bit too conservative. Shimano says to replace them at 0.5mm, but you can go safely until you see the dog-bone pattern, so I was sorting out the thickest ones to re-use to get a bit more life out of them.
The parts shortage (and price increase) is actually a pretty good argument for rim brakes.