I toured on 32s and did not see the disadvantage. I suppose on gravel, it could cause some problems, but not make it impossible. The biggest advantage of larger tires is the higher weight capacity.
People talk about comfort, but at 85 psi (42s) vs 95, I doubt there is that much of a difference.
If you want to try bigger, the caliper idea should work