If the inner tube isn't bulging out the sidewall, it's good to go. I've never had a wreck or even felt unsafe anytime I've flatted on a bicycle. Perhaps other have horror stories but I have never experienced anything remotely unsafe feeling.
You should consider where you are going to be riding and how a flat might impact you in those areas. If a tire is getting close to looking bad, I'll change it a week or so before I do an organized century ride or such.