I wouldn't have a problem with it and I doubt many people would notice. But why not actually participate? You won't get the full experience if you're just observing and riding along when convenient. Doing this difficult thing with thousands of people from across the world, all of whom qualified, and experiencing the controls and the hospitality of the locals--those are big parts of what makes PBP special. You'll hear the townspeople cheering, but you won't feel it, because you're not actually doing the ride.
The downsides you mention (lining up at controls and at the start) aren't a big deal. Much less of a hassle than riding 1200km in 90 hours or less. And if you're regularly doing long self-supported brevet type rides, then qualifying shouldn't be such a pain either.