Oh man! Once you start asking that question you are DOOMED by the answers you are going to get. No matter how much stuff that you carry with you, somebody will always convince you there is something else that you won't survive if you don't have.
For the kind of riding that I do (day rides, supported tours) all that I carry with me is the stuff that I need to fix one flat tire. I'm typing this from home so I've obviously always found a way to get back.
__________________
My greatest fear is all of my kids standing around my coffin and talking about "how sensible" dad was.