If you are not sprinting or doing a lot of out of the saddle accelerations, its not going to make much of a difference. Just for grins, I tried my favorite 3 hour ride (i.e. more endurance) on a bike that weighted what your did. I wasn't any faster, didn't really have any less fun the heavy bike. Well, until the end. I did run out of steam about 2.5hrs into the ride so it ended up taking longer - guess the extra weight wore me out (rolling hills).