I had a local Tri guy drafting me in 2015 and indeed, I was worried that I would be fined an hour. SO, I stopped and let him go. It was a little annoying but the local fellow on a training ride probably had no idea that he could be screwing me. OP knows he could be affecting someone's ride. As such, he should either not ride close to others in his informal PBP or do the qualifications to gain entry and then just skip the controls.
I have not figured it out but the controls add an element of excitement. There is a level of seriousness one does not often experience. Let me pick on Villaines. In 1995 starting in the 84H group, we arrived before the official open. They scolded us. I got my card stamped. They were holding us there. I had to pee. The control then was not at the Mairee. I was going several hundred metres up the road to the trees and a stern controller was stopping me. I said something like, Il faut faire le pee pee, tres mauvais, tres mal. He looks at me face to face in the eye, points to the bush. Et, faire le pis and Ne bougez pas!! So, I peed and then waited until 5:53 or whatever the minimum time was to that control.
Then, 20 years later in Villaine I arrived a little slower having been dropped and no other riders was there at the control. I was the only rider. Vide. I asked where I could go toilette. The lady stamping my card took me by the hand and walked me there AND back. I am not sure why this memory means something to me. Surely as an older person and arriving there pretty fast, I suspect she was showing me respect. But overall, it just seems that the French take cycling seriously and they accord us crappy randos respect that we might not deserve or simply that they appreciate the effort.
Last PBP had me having some medical issues and my focus was just enjoyment and finishing pretty good. I could not find my hotel in Brest and decided just to sleep at the control. No beds. I was freezing my ass off, tired, and sleepy. I just laid down in the dorm by the gate keepers to the beds. They asked if I was ok. I said, "yes".....just old and confined to a velo couche (recumbent). They laughed their asses off. Asked me my age. Suddenly a room appeared from nowhere. I slept 7 hours. Still sleepy but I let at dawn on Tuesday. A photographer got picture of me riding with my eyes closed. I remember that exact moment although I never saw the photographer. The sun was coming up over the roc and it felt good, I closed my eyes. It was a very intense moment. Had I not had a number on my bike, that photographer would not been able to find me and offer me the shot. It would have been a moment lost......had I just done the ride without registering
Skipping the controls for me is a huge miss.
Do it right. Respect