Situational awareness is key. That said, when I'm near The City, I tend to grab a motel room and enjoy the shower, the bed, the receptacles to recharge everything that needs it, the laundry, and the restaurant and possibly theater nearby. Since the bike is in the room, I don't worry about it.
Thieves taking bike and gear happens, but not very often. I'd already heard about the Hawaii theft; there may or may not be another case like that this year. So the threat level is pretty low, unless you're on or near a college campus or some other high-crime area.
What risk are you willing to take? What cost (financial, weight, inconvenience) are you willing to take to mitigate an unacceptable risk? You could take an 8 foot logging chain to lock up your bike; do you want to haul a metal box on your bike to lock up panniers? You might consider credit card touring, finding B&Bs (with bike-secure facilities), motels, or the like to stay in rather than camping. Expensive? Not as much as chartering an armored car to lock your bike and panniers up every night. Yes, I'm venturing into ludicrous, but the point remains that you're going to have to trade off security costs against acceptable risks.
Camera, wallet, cell phone -- those can vanish quickly, so it makes sense to keep them near you. That's why you see so much more discussion about keeping them secure. My Ortlieb bar bag carrying those goes into stores and diners with me, and I take it into rest rooms and my tent at night. Dirty clothes always sit at the top of my panniers, to discourage someone wants to go fishing in them (just like packing for the TSA to inspect my bags!).