FG gravel grinding may be a more recent phenomenom, but there is nothing new about fixed mountain biking...
http://www.63xc.com
Going back some years I did it pretty regularly on bigger clearance 700c fixed bikes with CX tires - a Redline 925, then a SE Bikes Fixed PK Ripper, and finally a Steamroller. Amazing how much fun it can be going slow. In fact, some guy literally bought my "Ripper MTB" out from under me at a trailhead after he followed me around the trails a little and became fascinated with it.
But I do agree with the others who mentioned that an older '80s steel MTB with front loader dropouts would be a cheap/easy way to get into it as long as you can deal with their longer wheelbase/touring geo, and come up with or rig a rear wheel to fit the bigger OLD. An alternative would be to look for a used fixed gear freestyle bike - since that died out they should be available pretty cheap...errr...inexpensively.
Good luck and have fun...