I'm currently in vacation and am using a Panasonic GM5 with 12-35mm lens and a 20mm pancake lens.
It easily fits in a jersey pocket. It's M4/3, which you said you didn't like. But it's what I use. I also have a tiny camera bag (because the GM5 is tiny) that I can sling across my shoulder and it doesn't distract from my riding.