I have a really simple approach to the front end of a bike: that everything from just behind the headtube forward is "sacred apace". Anything I add there gets real scrutiny. Is it absoluteely safe? Is it absolutely safe if I space on looking at it? Is it necessary? Can I make absolutely secure?
I've lived through a fork breaking, I've seen what can a happen. I used to cringe at what my mom decorated her HBs with. Then one day she did a face plant for no apparent reason at all. I'm virtually certain it was her flimsy bag. Cost her the embouchure of a really good flute player. She never got it back.
I gave up on handlebar bags many years ago. I do clamp on lights and computers. Other than that, my handlebars are (save CX levers on one bike) clean. I'll admit I'm pretty casual with the drop-on Ortlieb panniers. Them coming off has never caused any issues more than requiring me to stop. And if you fasten the strap that runs over the top, nothing can get into the wheel.
Ben