What are the gravel roads like in your area? Sometimes uncompacted large-aggregate gravel can be a little bit like what you're describing...
...but in most areas this is fairly uncommon. Oftentimes the big chunks are compacted, or held in place by smaller aggregate or organic material. "Gravel riding" is also a phrase that used pretty broadly to refer to all kinds of road riding that may include unpaved surfaces, so things like dirt and grass roads are included in the discussion as well.
Aside from potholes, this gravel road was smoother than most paved roads in the same area:
What "gravel riding" means is also highly regional. People in different places build unpaved roads in different sorts of terrain for different reasons, and they have different material available to build it from. The rural unpaved road network of Iowa is quite different in a ton of different ways from the various mazes of mountainous logging roads in Washington.
But no, the point of gravel riding is not to bounce around on marbles. That's something that incidentally happens, sometimes. Mostly I ride gravel because there are lots of fun gravel roads to ride in my area, often with excellent scenery and views.
I can't ride this awesome road without riding gravel:
