I can understand why bike-packers get fully loaded up, but I'm amazed what some people lug around for a relatively short ride. My kit for a century ride consists of:-
1. Dynaplug puncture repair kit
2. Mini-pump (attached to bottle cage)
3. Multi-tool
4. 2 large bottles
5. Phone, cash and credit card
6. Energy bars, gels
7. Packable rain jacket (if it might rain)
I just carry those small items in my jersey pockets.
I do have a very small saddle bag that I rarely use anymore. I used to carry a spare tube and tyre levers in it, but after not using them for 3 years I decided it was redundant.