Experiment. personal taste is more at play here than hard data. Find the widest tires you can fit on your rims and give it a try. Play with different pressures. I am not convinced that three is all that much more comfort gained from soft 38s over soft 32s .... but it depends upon the roads you ride and your definition of "comfort."
As for speed .... if your wheels and tires weigh more, they will feel a little more sluggish when accelerating (when accelerating forward, for the physics sticklers) but you might not mind or might not even notice. How important is speed? Everyone has a different answer.
I think the last line about subjective interpretation is all-important. Everyone has different interpretations of how their ride should "feel" (in quotes because it is a combination of factors and purely subjective---indefinable) and for some, the "feel" is fine with wider tires, heavier wheels, harder tires, whatever .... So the only real answer is "Try some stuff and see for yourself."