Unless you're racing, you should ask yourself how much you want to push the envelope. Out on a road that is open to the public, you have to be able to deal with whatever you encounter. Cars in your lane, unexpected debris on the road, a stopped vehicle, etc. Chances are, if you are riding at the limits of what you and your bike are capable of, you are not going to be able to react to the unexpected.
When I was riding a motorcycle, I would never for a second try and ride on the street the way I would on a track. I feel the same way descending a hill on a bike.
That fear your experiencing is likely based on your brain knowing that you aren't fully prepared to deal with whatever is around the next bend. Listen to it.