I've found this to be true also. I've got a ~30 minute climb near my house on gravel, and I've compared my power output vs similar paved climbs and its always ~20-30 watts lower, I'm thinking its due to being bumped around a bit.
I've found that the best way to climb on gravel is to use a bit of a lower cadence and bit higher gear than I would normally use. For some reason it seems to make the bumps feel less bumpy.