I have a new Giant TCX, and have spent the last week covering my usual rides, which includes relatively smooth gravel and lots of tarmac. I've taken it on some soul-crushing climbs and warp-speed descents, including getting caught out in the rain for a couple of hours.
Great bike; if I was booted into the afterlife, and given the choice of only one bike to take (into Hell obviously..) then this bike would be right up there for consideration.
Bottom line: this bike isn't as 'fast' as a full-on road racing bike, the difference most notable on the climbs. First off, at 19 pounds (with pedals), it weights at least 2 pounds more than an equivalently-spec'd road bike. Unfortunately, most of this extra mass is concentrated around the wheels and the fork. On a full-on race bike with rim brakes, you can easily spec 1,500g wheels, and lighter and faster 23mm rubber. And this is for clinchers - tubular wheels are much lighter again. However, my new Giant has discs and (good) 32mm tires, which is the cause of most of the ballast.
Sure, the fat tires are more comfy on gravel, but on the gravel I frequent, 25mm tires ride just fine. I am light, and careful, and if you go fast enough, every surface 'smooths' out.
The discs are the other cause of the weight, as disc-specific wheels and forks simply have to be bulked-up relative to rim-brake equivalents. The extra stresses caused by discs, particularly on the fork and front wheel, requires this. So the fork has to be made stronger, which in addition to extra weight, means it rides with less compliance.
However, if you want an all-rounder, and need discs for descending in the wet, then a current 'cross/gravel type bike would make an excellent choice.