I think it's a little unhelpful that we refer to anything that isn't pavement as "gravel." I hate riding on actual gravel - pea sized rocks, bunch of them, especially when it's deep. We have a lot of hard packed dirt roads here, and these can be fantastic to ride on.
Like others have said, it's about scenery and traffic. For me it's also about having more places to ride, and about being able to make a loop out of what would be an out-and-back otherwise. I think we only have 5 (?) roads that cross the Cascade Range, and we have a lot of roads (that mostly serve hiking trails) that are paved for about 10 miles, then turn to dirt. We also have a maze of dirt roads throughout the mountains, so I can ride those first 10 miles, and instead of just turning around, I can keep going. Lot of beautiful paved roads, but eventually you get to the point where you've ridden most of the ones that appeal to you.