I'd say 40mm tires will be wide enough for actual gravel roads, but on singletrack trails you might be underbiked (which isn't necessarily bad, you'll just be a little slower).
Also, rider weight is a factor - if your friend with 42mm tires is 30 lbs. heavier than you, you can run your 40mm tires at roughly the same pressure (maybe even lower) as your friends 42mm tires and have roughly the same comfort level.