If you believe Jan Heine (
https://www.renehersecycles.com/wide_tire/), there really isn't such a great disadvantage to running fat tires, or even fat knobby tires. The important variable in tire efficiency is the flexibility of the sidewall, rather than the size. A high quality 2" tire rolls just as fast as a 1" tire. Now, I don't really want to get into arguing about whether Heine is right, or what his motivations are (he is the only one selling the tires he recommends, after all); but there's no denying that he sells some pretty expensive tires, and a lot of people are using them for long distances.
Last season I rode a lot of miles on 53 mm tires-- a whole SR series, two fleches, a 1200 km brevet, etc-- and if the tires slowed me down, it wasn't by much. With such fat tires, you don't hesitate to ride on rough or even unpaved roads. While the fat tires seem to have a distinct advantage on rough roads, they don't seem to have any disadvantage on smooth pavement. Seem, I say-- because it is hard to tell what's slowing you down. More readily apparent is the overall comfort of fatter tires. After last Saturday's brevet, a friend remarked that he felt like he'd been pummeled in a bar fight. I didn't; and I weigh more than he does.