You should be scared. That is what keeps you on your toes. But don't let it rule your life. Go out and enjoy it.
I think about it from time to time Scares me a bit. I get most scared when I see a car pass, look up and they are staring down at their cell phone. I know they didn't see me. It was just luck that they did not veer into me. But I get over it and enjoy the rest of the ride.