I use my Niner RLT RDO as both a road and gravel bike, just swap wheels. If all you ever ride off pavement is nicely packed dirt/gravel, you can easily do so with 32mm tires. However if you encounter sand and ruts, bigger is better. I run 38s, and would not go narrower for my local trails. Hit several sandy spots just this afternoon, I can manage them reasonably well with the 38s. I think most road bikes max out at clearance for 32s right now.