I am ten years older than you. Three years ago I went from a from a 19 pound road bike to a 34 pound touring bike as my daily rider. It definitely effected my speed and acceleration.
I am talking at least three miles an hour.
My legs are stronger than ever and I am very fit. It would take a big strong man to be fast on my touring bike.
I recently finished a custom build of a similar fitting bike on a non suspension mountain frame and got most of the speed back.
I went to the heavier bike because of wrist issues and multiple surgeries. I have no good cartilage in my left wrist and had to go more upright to continue riding a diamond frame.
Not to dis your son but he has no idea what being your age is like.