You could dip a toe into fixed riding by riding single speed. You can determine your gearing by selecting a chainring and cog on your multi-speed bike, and not changing gears, when you buy or build a fixed bike, you'll know what ratio works then. 3/32nds or 1/8th? Drop, flat, or pursuit bar?
What's your tolerance for braze-on fittings that become superfluous, like shifter hardware and cable stops?
My only gripe with the modern, commercially available fixed-gear street bikes (track bikes are really neat, but aren't designed with braking in mind), is that they're mediocre welded chrome-moly plain gauge frames with Sunlite grade cranks, rims, tires... they're cheap to hit a price point. Lighter and more responsive frames start around $400 new, but still have the cheaper finish components.
Read through Sheldon Brown's
page about converting old bikes for fixed gear duty.