WizardOfBoz

Skipjacks, you've hit on a key point. Is the buckle breaking that was reported based upon a sufficiently powered test? That is, did they test a sufficient number of helmets to characterize the variability in the sample.

Further, was the testing of the buckle representative of normal and emergency (that is, when the helmet is needed) use?

I'd like to know how they tested the buckle, and whether that test was relevant. If they arbitrarily tested the buckle for, say, 100 lbs force - does that matter? Would a 50 lb test be more than adequate for normal use?

I would also wonder if the CR review was as thorough as the Bontrager tests. I suspect not. See here: https://www.trekstop.com/product/bon...l-357580-1.htm

And last, if the Bontrager tests are accurate, the newer WaveCel helmets are significantly better in a crash. If the CR tests aren't directly relevant, then the promise of newer, better technology is dashed by poor science and sensationalism.

Ralph Nader's criticisms of the Corvair were all proven false. But it was successful in killing a nascent small-car industry in the US. Likewise, there were dubious claims of small car rollovers for a specific Japanese small SUV (can't recall it). Pretty much all irrelevant, and no statistical ever showed that the rollovers happened in use. I'm pretty skeptical of the CR review.

WizardOfBoz

Again, not sure I trust the CR test methodology when it comes to a broken buckle.

The most recent VA tests I could find have a now-discontinued Lazer MIPS model, the Cyclone, having the best rating of 10.5, with the Bontrager Wavecel is in 2nd, having a rating of 10.8 (lower is better). But its pretty shocking to me that the published VT tests don't report standard deviation. 10.5 vs 10.8 with a standard deviation of 10 has no meaning. 10.5 vs 10.8 with a standard deviation of 0.05 starts getting to be a real (significant) statistical difference. I'm gonna go out on a limb here and suggest that a 3% difference in destructive test results (which are often pretty variable) is probably not very significant.

There are eight helmets within 20% of the low value in the VT tests. All are Wavecel or MIPS. There are also helmets that have a rating 250% of the low value (including a different Lazer model).

As far as the top helmets, a key point besides statistical significance is clinical significance. If 0.3 is a real difference, how does that translate to probability of head injury?
What I was interested in was a model of the test results that predicted the percentage of people getting concussions or more serious TBI.

There is a published analysis of the Wavecel helmet that attempts this. It was coauthored by guys that invented and patented the technology. But the results look pretty convincing that it is better technology. They used a Scott Arx MIPS helmet and compared it to a non-MIPS (Scott Arx). They also took Scott helmets and replaced MIPS with their Wavecel technology for comparison. A quick review suggests that the results that the Wavecel people got were in substantial quantitative agreement with the VT tests (I would want to do more analysis before I swore to this). In addition an overall score, they provided the standard deviation and statistical significance of test differences. Note that the standard deviation found in this paper was on the order of 1-2 %. If the reported values WERE standard deviation (and not standard error of the mean) then the 0.3 difference above is likely statistically significant.

What is clear is that both Wavecel and MIPS appear to improve your likelihood of avoiding injury from head trauma. All of the top models were one or the other.

Evaluation of a novel bicycle helmet concept in oblique impact testing
Emily Bliven, Alexandra Rouhiera, Stanley Tsai, Rémy Willinger, Nicolas Bourdet, Caroline Deck, Steven M.Madeya, Michael Bottlang
Accident Analysis & Prevention, Volume 124, March 2019, Pages 58-65

WizardOfBoz

I come out on the opposite side. The helmet retention test looks unrelated to what one would see in an accident. They drop a nine pound weight two feet and see if the buckle breaks or if the chin straps pull out of the helmet. The certified CPSC labs do the same test, and did not see a broken buckle. The CR lab is not, apparently NIST certified. And I agree that there's nothing wrong with ranking and comparing IF THE RESULTS RELATE TO NORMAL USE. But if I used a slingshot to shoot a 1/2 inch steel ball into a helmet to test it, I think it would be irresponsible to report the the steel bal got through some helmets and not others. Especially in such a prominent place as CR. Because you'd probably find urban style helmets superior to road helmets for this test (the ball could go right through a ventilation crack). Whereas the VT and CPSC tests generally show that road helmets offer better safety.

I think I agree that the CR people did the tests that they say they did in the manner that they say they did them. Just not sure that 1) the chin strap test is relevant to reality, and 2) not sure how CR got these results and the CPSC did not.

https://www.bicycleretailer.com/indu...s#.XR_SJOhKhPY

livedarklions

If you look at the VA Tech rating system, it's actually clear they don't consider small point differences significant. It's actually a five star rating system, and they recommend any four and five star helmet. My point was that the Bontrager helmet was near the top of their list, not that it's some sort of knock that it's third. I was contrasting an excellent VTech rating with an unacceptable CR rating. I bought the Cyclone, btw, it was remaindered so I got it for $30 right after the VA Tech results were published. Nice helmet.

I have to say, your description of the Wavecell study convinces me their claims are a lot of hooey--that just shows that Wavecell>MIPS>non-MIPS for exactly one specific make and type of helmet. Generalizing from that to blanket statements about the superiority of one tech over another is absurd.

Sy Reene

The point is though, there are 81 helmets in CRs review database. I wouldn't buy one of those 3. Can't say if CR's test is better or worse than CPSC, just that it's different. I do find it interesting in the case of the Woom, that this was said:

"I am very surprised about the test findings since all helmets had passed all regulatory safety tests at an accredited independent test lab in 2018 very successfully before we launched them in the U.S."

I wasn't aware that CPSC certifications are farmed out to independent test labs. What does passing "very successfully" mean? We might all know more if test results were published by CPSC, but I assume it's a simple pass/fail system. Even so, a broader picture of how many submissions did fail, and for what reasons, might be enlightening.

WizardOfBoz

I understand your point. I completely disagree with it because I question the CR methodology. I'd have no problem buying any of those helmets. Especially the Bontrager Ballista MIPS, which was the third most highly rated helmet on Va Tech's list.

My point is: is the buckle breaking in an unaccredited lab (CR) credible or relevant when the buckle didn't break in either the CPSC test or in the Va Tech testing? And given that Trek could not replicate the buckle breaking WITH THE SAME TEST? Also, is the buckle break test relevant to helmet function? Do buckles routinely break in accidents? Is the stress put on the buckle of the same order as that seen in an accident.

My job is to write models of human disease. This involves both statistical models (parameter estimation, estimating statistical significance of different results from different treatments) and physiological, mechanistic models. This type of model would keep track of, for example, glucose in the blood, extracellular fluid, stomach, gut, and so forth, and would keep track of glucose flux into muscle, adipose, the brain and the liver. The various numbers in this type of model can be gleaned from science journals and books and such. These models are used to make decisions. So I'm really sensitive to folks using information in the proper way. Before I would condemn a product on the basis of the reported CR tests, the results of which are at odds with several other labs, I'd want to know more.

How many identical helmets did the test the buckles on?
Did they lower the weight used to find a threshold value for buckle breakage?
And so forth.

It's ok - we can agree to disagree. I'm just saying that when one lab gets a different result from several other labs, and the mfr (for at least one helmet) can't replicate the result, I'm deeply suspicious.

Sy Reene

I interpret that independent accredited labs, mean that each manufacturer pays (ie is the client of) the lab to have a test performed. Conflict of interest? We basically have CR's lab, vs a paid (by Trek?) lab, and Trek themselves.

If CR replicated the CPSC test, it would be absolutely pointless. I would applaud the test being different to some degree. Otherwise really, what would be the point?

VA Tech doesn't test buckles, just impacts. And yes, I would assume the ability for a helmet to stay on your head in a crash is relevant.

As for the mfr, they can't exactly come out and say "yeah.. from time to time our QC department found this to happen."

I don't have any 'suspicions' as I can't identify a winner or advantage to CR by identifying a product as a don't buy.

FYI.. CR shows that the Bell 4Forty had a similar finding; amongst many many other Bell helmets that had no issue. Just like Trek.

WizardOfBoz

Sy Reene,

Not arguing your points - all valid.

But once a real issue is identified it would be liability suicide to replicate the tests, find the same error, and say "nothing wrong here". So I think Trek does have an incentive to report honest results.

But we have different takes on the information. We are inferring one thing (whether the helmet is safe in use) from other observations (a buckle breaking, or not). I'm looking at the superior head protection as measured in the Va Tech tests, and the fact that the mfr hasn't observed broken buckles and says it can't replicate the issue. Your taking the data from CR, noting that they have an incentive to be impartial, and ruling out those helmets. Truthfully, there's no way to definitively say one way or the other without a lot of field tests with the specific helmet, so as to get a statistically sufficient sample size.

But you have some compelling points.

WizardOfBoz

Ok, the article I saw covered the Bontrager MIPS Ballista, the Woom, and the Morpher flat folding. So now they're including the 4Forty helmet?

But in looking at this again, I noted that CR reported helmet buckles breaking in 1997, and the CPSC was unable to replicate their findings. Same thing in 2014. And now in 2019. In science, repeatedly publishing stuff that others can't replicate is pretty conclusive evidence of either incompetence, poor communication, or outright fraud. In any case, repeated publication of irreproducible results strengthens my feeling that their tests were flawed in some way. I'd say that this casts some shade on CR's reputation and that this whole series of buckle reports more severely threatens CRs credibility than Trek's.

Not sure what the "Bicycle Helmet Safety Institute" is, but they claim to be independent and consumer funded and on their website they reported some stuff that supports your argument (for Morpher and Woom) and some that would seem to support mine (For Bontrager). On July 3rd of this year they wrote:

"Consumer Reports has published an article on their web page announcing that three helmets have failed their testing:

• Bontrager Ballista MIPS
• Morpher Flat Folding
• Woom Kids

CR's tests are essentially the same as the testing for the US CPSC standard. In the case of the Morpher and Woom helmets the failure to pass impact standards probably means both are marginal for impact performance and would possibly pass in some labs but not others. We would avoid both, since most manufacturers build enough margin into their designs to accommodate both quality control issues and lab variations.

The Bontrager buckle failure follows a number of similar failures over the years in CR labs in 1997, 2006 and 2014, and is of less concern. The CPSC buckle test is a severe test, and the buckles Trek uses on their Bontrager models are standard ones that seldom fail in other labs. On occasion a manufacturer uses a slightly stiffer chin strap or the geometry of the strap system results in less stretch during the peak force of the test, stressing the buckle more. Trek says that "independent lab testing conducted after Consumer Reports' notified Trek has been unable to replicate Consumer Reports' findings." Trek has no reports of any Bontragers failing in the field. That has been the case in the past when CR reported buckle failures on other models, as we reported most recently here in 2014.

CR has also published an article about helmets they have been able to buy online from Amazon, Sears and Chinese sites that do not meet US standards. We have been pointing that out for years and noting that the cheapest deals from Chinese sellers--even on US branded helmets--can be knockoffs that lack the internal reinforcing and quality of construction that make the real version available from US retailers able to pass standards.

CR also announced again that they are working on a welcome update of their 2016 helmet ratings. "

livedarklions

I think part of the problem is that it's impossible to predict with any certainty the kinds and intensities a strap and buckle might be subjected to in a real-world crash, so it is really difficult to know which testing regime is more appropriate. What I am seeing here is that CR uses a test or tests beyond those used by CPSC, and I'm not clear whether Trek used the cr methodology or the CPSC.

if Trek used a different methodology, "failed to replicate" is somewhat meaningless.

i probably wouldn't buy one of the non-recommended helmets on the cr list, but only because I would rather err on the side of caution.

WizardOfBoz

Exactly. The relevance is one thing. The repeated failure of other labs to replicate CR's buckle breaking results (in 1997, 2014, and now in 2019) also factor in to my skepticism about whether the test was done properly, and about whether it means anything at all.

I guess my point is that, in a multivariable, nonlinear, highly variable, and (as your note points out) largely unknown system like prediction of head injuries from a buckle break which was only seen in one of three tests, I'm not sure that not buying the Ballista IS erring on the side of caution. Before the CR review, it was rated as one of the safest helmets. It may still be - and not buying it means buying a helmet with a lower rating in impact. If the impact test is more relevant, or the buckle test is not representative of a real crash, not buying that helmet and buying one with a significantly lower impact rating (but a functioning buckle) means you'd be lowering your safety.

The CR methodology IS published, and IIRC is similar to the CPSC test. I'm sure that Trek tried to reproduce the exact test (it's basically dropping a 9 lb weight 2 ft, using a chin-shaped hook on the strap, while holding the helmet body steady.) So if they're using the same test and get different result I want to know why before I place any reliance on the data.

This is also not theoretical to me. My best friend had a horrible rock-climbing accident. Nearly died, and possibly would have been better off if he had. From a university professor and world expert to not being able to form a sentence. Helmets are important.

I think that the two that had lower impact scores, ok, I'd probably not buy 'em. I'm no big Trek fan (I have a Trek bike I like, but I'm still pissed at 'em for what they did to Greg) so that's not it. What I really don't like is people making decisions based upon specious, irrelevant, or possibly misleading data. I see it all the time in Pharma. Don't like it there either!

canklecat

Every Bell helmet I've owned and handled has pretty much the same buckle. I wonder what the difference was in the tested model that failed?

Personally I'm not a fan of the Bell buckle. It's difficult to operate one-handed, especially with sweaty hands. But it seems very unlikely to fail and no more prone to breaking than any comparable buckle.

I'm more concerned about how much of the skull a helmet covers. When I've examined my own previous helmet after a crash, and that of a fellow rider who crashed hard last week and slid headfirst into a curb, I'm concerned about whether the typical roadie helmet extends far enough to protect the occipital region of the skull. After watching that crash directly in front of me, and reviewing my handlebar video, it's pretty terrifying how quickly things go wrong and how little we can do about head knocks, other than a few ounces of plastic and styrofoam mushroom shaped beer coolers on our noggins.

I've worn a Bell Formula MIPS for the past year and like it overall. It's comfortable, well ventilated, lots of attention to detail including the way it channels sweat away from the eyes, and full hard shell laminate over the softer EPS foam so it resists chipping.

But I've been concerned that it doesn't extend as far down the sides and back of the skull as my previous Bell, or as much as most mountain bike helmets. To some extent the current Bell roadie helmets, from the Formula to the Z20, appear to be leaning toward minimalism while meeting safety standards. They don't look so much like plastic noggin mushrooms. But I wonder whether I'm giving up protection for cosmetics and aero benefits, when I'm not a racer or even particularly fast.

So I just ordered a POC Omne, which looks like it should cover just a bit more of the skull. We'll see after delivery this week.

I'm also tempted by the Smith helmets with Koroyd crumple protection. Same concept as WaveCel, but a bit lighter. So far the folks I know who've bought WaveCel helmets like 'em, but they're fairly casual riders -- not pushing hard in midday summer heat. Users who've tried both say the Smith roadie helmets with Koroyd breathe just a bit better.

livedarklions

Half empty vs. half-full. I would want a helmet that was roughly comparable on impact without the identified buckle problem. I see no basis in the arguments for finding one testing regime is better than the other. And no, I don't give Trek the benefit of the doubt on the "failed to replicate" language . To me, that sounds like a phrase cleared by the marketing department as it's quite vague but sounds specific.

I've got no dog in this fight, btw, but I do think Trek has been over-hyping their helmets. I'm sold on either MIPS or Wavecell at this point, but don't think there's any significant difference between their effectiveness.

WizardOfBoz

Like you, I'm shocked - shocked - to know that a mfr is overhyping their product.

I come down to the same conclusion on MIPS and Wavecel. I've read the research papers, and the cluged-together Wavecel helmet (Scott Arx with the MIPS innards replaced by wavecel) beat the Scott Arx Plus MIPS by a pretty significant margin (about 50% higher, or worse, rating for the MIPS). . In this test (by the Wavecel inventors) the difference between their Wavecel and the Scott MIPS helmet was consistent with the Va Tech data so I don't think its been "adjusted". Other MIPS helmets (including the Ballista) were very close to the Wavecel Specter. And in fact the Specter beat the more expensive XXX. Both technologies improve safety I think, and are better than the styrene foam tech. Wavecel may have a slight edge, but its not the step-change over MIPS that was promised.

This whole thread has been useful for me: its allowed me to do some research on these helmets. On thing that came out was that the Wavecel helmets do seem to give a sense of cocooning the head better, but they can slip down and mess with your sunglasses.

I just thought of an interesting point about that buckle breaking. There could be a scenario where a helmet gets caught by a bumper or truck mirror, and if that buckle doesn't break you end up strangled or with a broken neck. So I think a compelling case could be made for the buckle breaking (if it really does break) being a safety feature.

livedarklions

There's almost no way to build a decent database on exactly how the head gets impacted in actual accidents because so few of them are documented to the point of knowing what the rider collided with exactly, when in the fall occurred, and the speed of the actual impact (let alone g force), so all tests are guesses.

@Witterings and I had an exchange where we discussed helmets saving us from strikes by low-hanging branches, not sure if there's a scenario where the helmet could get stuck in the branch or not, but it sounds vaguely possible.

Like I said above, I don't think a test that showed that a single helmet design worked better as a Wavecell helmet than its MIPS version proves anything about the general superiority of Wavecell over MIPS, and VA Tech data shows the technologies virtually tied when tested over a much larger number of models. I wouldn't buy the Wavecell because of the cost when I can get equally safe MIPS helmets for much cheaper.
BTW, I've posted before that I consider cheapness to actually be a safety factor because I think people will consider the cheaper product to be disposable, and are more likely to discard it after a crash.

WizardOfBoz

Concur on all points.

Clearly (as my overly verbose notes prove) you can go down a rabbit hole of detail and never (to mix metaphors) come up for air.

It certainly would be difficult to build a database that would be meaningful. I think you'd have to have every helmet with a built in set of accelerometers and data loggers. After every accident you'd send in the helmet to the mfr, along with info on injuries. Then you could get somewhere with retrospective analysis. That could happen, as you probably could engineer such a system to be the size of a marshmallow at the mfr cost of about a buck or two. I didnt' look but I think that the massive effort to reduce football brain injuries may have yielded something like this. But for now, I guess we could send the helmets in which would allow estimation of the intensity of the blow using helmet damage, even without data loggers.

I was looking at the test stand setup (below) and was thinking: what if a helmet, in protecting the brain, made it more likely that you would break your neck? Oy.

The five star rated helmets are all MIPS or Wavecel. There are helmets in that class priced at 75 bucks (even though that top-rated helmet isn't available anymore). I'm probably going to ask for a helmet upgrade for my birthday. Probably the Wavecel Specter, because its cheaper than the Ballista MIPS. Of the road style helmets, the Specter is, at 150 bucks, among the cheapest of the five star rated helmets. If you want an MTB style (that is, with a visor/brim in front) there are cheaper alternatives. You make an interesting (and I believe very credible) point about cost vs safety. The best helmet that you can't afford, or an expensive helmet that you can't afford to replace after its been through an accident, is less safe.

BTW, people don't realize this, but after you've had an accident in your car, you should have the seatbelts that were worn serviced (e.g. the pretensioner unit) or replaced. A lot of shops won't say anything. Insist upon this.

https://www.bicycleroots.com/blog/wp...esting-725.jpg