Obviously a lot of drivers are going to have this attitude, and a certain fraction of them are going to be enraged and react dangerously to a bicycle slowing them down. I went through all of this before deciding to go car-free for a couple of years. The bottom line I came to is this: anyone who strongly objects, including and specifically with the reasoning "car-sized objects going 55 and small bicycles going 15 just don't mix, physics always wins" is simply wrong. Legally wrong, morally wrong, logically wrong, and wrong in practice.
In this country, a legal right is fundamental. As long as you're going about your legal and rightful business, excepting extraordinary situations, you do not have to justify your free exercise of your legal right to anyone including anyone in authority, let alone the random person on the street. If that person has a problem with it, it's their problem. If by their actions you are forced to face extra risk, they are in the wrong legally and morally, and it's their obligation to alter their behavior, not yours. If they refuse to do so, if they rant and rave about the cyclists, then they are the ******-bags.
Therefore, as long as you're operating legally and you respect common courtesy enough not to exaggerate your rights and push the boundaries, you have the right and are right to be on the road of your choosing. And in practice it's not nearly so risky as the inexperienced or ignorant will suppose - because the laws governing these activities are designed that way for good reason: because over years and decades they have been found to work.