My guess at the title question is that in general people assume that other drivers are just as bad as they are. In other words, they fear themselves (or other drivers like themselves).
Interestingly, the body question more about why pedestrians don't ride. My guess there would be that they choose to walk for simplicity. I say more power to them on that front!