I’ll also go with the pacing and conditioning causes, not fuel. I’m about the same size as you, I also have oatmeal and a banana before most rides. That’s usually enough for about 60 miles and 3000 feet. I carry a Clif bar, if the ride is harder, longer, or I’m less fueled starting out I will eat it.
For rides over 3000’, I use my heart rate to pace myself on the early climbs. Assuming it’s not unusually hot, if I keep my HR in upper 140s or lower on climbs then I can go the distance and finish strong.
I’ve also been riding six years now, it makes a difference. First year I definitely had to eat during a ride. Eventually reduced to just a mid ride Clif bar and in the last year or so only intermittently if I start to feel hungry. If I’m going beyond sixty-ish miles, such as a century, then I make a point to nibble a bit at each of the rest stops.