Of course 40mm Marathons are going to feel slow. There is no reason to use those tires. They offer a little more puncture protection but the ride feel is terrible.
Switch to 40, 42, or 45mm tires with a supple sidewall and you'll prefer them for all types of riding, even pavement. Compass, Soma Supple Vitesse, WTB, Grand Bois, whatever. They are leaps and bounds ahead of the Marathons which are overbuilt traditional touring tires.