I never went the "only CX bike" route, but I think I could if I wasn't mountain biking too. The bike would do a lot of things really well and you could pretty much ride anywhere. The gearing is a bit different on a CX bike, and the geometry is too, but all that can be worked with to provide thousands of "anywhere" miles.
I'm currently selling my Rivendell road bike and replacing it with a different road bike; I don't really think I could be without something that I can just leave the house with and enjoy a thirty mile solo ride after work. I need a bike that does that just for sanity...but a good CX bike could do that too. I can see simplifying that way.