people tend to focus way too much on rolling resistance as the sole factor in cycling.
Taking a fast sweeper at 35-40+ mph in a descent, or trailbraking hard around a hairpin at 25 mph, you will quickly learn to appreciate front end control, and that tires that give a good rolling resistance (eg, gp 5000) will give far inferior feed back to a supple tire with latex tube. But sure, if you the money, then go all latex front and back. I have a lot of bikes cost of tubes is a factor for me. While latex tubes can be patched, they are more diffcult to patch due to the slick nature of the tube, and I've patched latex tubes go leaky on me after a while, forcing me to change to a new tubes, and latex tubes cost over 2x more then a light butyl.
Personally, if I'm riding in a group, even in a fast group ride averaging 27-28 mph, I almost never worry about rolling resistance as a factor holding me back. But the ability to take a corner at a higher speed than the guys around me is quite satisfying and energy saving. Rolling resistance will mean nothing if you get dropped coming out of corners.