I like gravel riding because I'm a roadie in an area without many paved roads (well, ones that are safe to ride) Gravel biking is like road riding without the pavement and I love it.
One point of concern, if you are afraid of going fast on a road bike, gravel riding can involve the same speeds but on loose surfaces. The gravel races around here will be running speeds in the high teens/low twenties (MPH). Maybe look at the "event" side of things more than the "race" side?
No matter where or what you ride, I'd suggest learning some basics for bike maintenance and repair. Simple deraileur adjustments, changing a tube (yes, even if you run tubeless) and so on can really change a day. It might happen only very rarely, but being able to limp home is 99% of the time going to be better than waiting for a ride.