Why would I?
In my area they don't seem to be a bother. Those I've seen on the MUP and elsewhere are all being ridden at a leisurely pace. There was one guy on a really heavy fat-tire e-bike that tried to stay with me on a climb, but he wasn't able to keep up with me. Don't know why, he never caught back up in the other six miles I had remaining for the day.
I don't know if any of them have been the type of e-bikes that should not be called e-bikes.