Dirt and gravel road riding on skinnier tires
I started riding on gravel and dirt roads in earnest about 10 years ago after moving closer to a very nice dirt country road area. I started on a MTB bike with 2.0 inch tires and have progressed down to 32mm tires on basically an older road bike. Sometimes really loose soil will cause rideability issues but usually the roads are hardpan or so compacted by auto usage that it rides like a rougher paved road. Even in winter I ride this ride after a rain by letting it dry for 8 hours or so. Otherwise the mud will bring you to a halt. My favorite width for dirt/gravel road riding is 40mm which is very close to a 1.75" tire size. I'm 6ft 5in tall and my road bike has a 67cm seatpost length. I usually buy older 27"x1 1/4" steel road bikes with fender clearance and switch over to 700c or just use the original 1 1/4" wheels. 1 1/4" tires are the equivalent of 32mm which I have found ride just fine on dry dirt and gravel roads. Switching to 700c has allowed me to run up to 40mm tires on older 10 speed road bikes. There are many options for the taller rider including bikes from the past. A little thinking outside of the box might open a very nice bicycling experience for you. I've hit speeds of 20mph on dirt/gravel roads fairly often with no control issues. It might be that you are very strong and causing the bike to swerve from side to side with pedaling effort. One also cannot fool with turning the handlebars much because it is easy to slide out on dirt. Steady pedaling with no great power surges in your stroke and a steady hand on the handlebar should steady your bicycle. My wife and I have ridden thousands of miles on the local dirt/gravel roads with no accidents for over 10years, knock on wood. Good luck with your new bicycle.