On long brevets, many randonneurs will answer the "how far are you riding" question by just describing the distance from the last control to the next control, like "about 60 miles", instead of the real answer, like "100 since this morning and another 110 miles today, then 150 tomorrow". That's to avoid the disbelieving and unnecessary chatter that would follow. We have miles to ride, not time to be dorking around with people.
So, yeah, being an 'older rider saying I've ridden 35 miles and getting astonished remarks, when in fact I've put in a century before lunch and will get in another before sleep, it's kind of weird.
My take on it is, people have no clue what they are actually capable of doing. When they encounter someone who's approaching the limits of capability, they are blown away. It is what it is.