A few points:
First, how toxic the venom is doesn't mean anything unless it's coupled with the dose the snake delivers. I think I remember reading, some years back, that the only rattlesnake in the U.S. that can kill a 175 pound man is the eastern diamondback, because it's big and has correspondingly large venom sacs.
Second, physical barriers can protect you. Put on blue jeans and wear shoes in camp, if that's where you're most worried. Save the shorts and sandals for places where you're not so scared.
Third, back out of striking range. Obvious, I know, but it needs to be said. Cowardly? Maybe. Safe and rational? Definitely.