A couple or 3 years back, a contractor's truck mirror bumped my tricep, and folded in. I'm sure it was intentional, the truck swerved on the country road I was on, no other traffic, broad daylight. Even if I'd seen it coming sooner, going in the ditch was my only out. At the time I was more mad than scared.
It was later the same season, though, on another nearby road that I did have a good scare- I was going pretty quick coming up on a one-lane bridge over a creek, with guardrails. An oncoming car going way too fast appeared, hit the brakes hard and too late, started fishtailing as we crossed paths on the bridge. Luckily I didn't get sideswiped and squished into a guardrail.
In both those cases there really wasn't anything I could have done. I always say, it's the one you don't see that gets you. If you don't have confidence and can't accept the risk, it's time to cool it.