I was thinking about the balance between a camera case that is safe and one that is convenient. If it takes a lot to deploy one would probably be inclined to take fewer pictures. In that case (pun intended) I could see a handlebar bag being set up as a padded camera bag with some egg crate foam thick enough to cradle the camera. You lose the carrying capacity of the bag for other things but have a convenient location for deployment.
For my Olympus TG Tough I use a handlebar feedbag. Even with the fish eye it fits secure but deploys easily.