If you aren't drinking anything on a ride of more than 20 minutes duration then I'd have to strongly suggest that is a part of the problem.
And if you do any hard accelerations or climbs or just riding at a very high effort for long periods then carb replacement is a thing you need to consider too. Whether it's by what you eat or drink. And carb replacement needs to start very soon after the ride starts, IMO.