Statistical Analysis of the Rokslide Drop Tests

[Bado20]

I was curious about seeing if their is a statistically significant difference in drop test performance between certain brands and the overall market. I conducted an analysis of the drop testing data and figured I would share my findings with everyone on here.

TLDR: Trijicon, SWFA, and Nightforce perform better than the broader market at statistically significant levels.

 

[Bado20]

PDF version

 

[4cMuley]

Someone has a touch of the ‘tism

Haha seriously though, this is actually awesome. Thank you!

 

[Harvey_NW]

I think the Maven RS1.2 results should be separated, as it's a known outlier.

 

[Bado20]

I think the Maven RS1.2 results should be separated, as it's a known outlier.

You should only remove outliers if you suspect that it is from poor testing or bad sampling. Unless you think Form didn’t do the test the same way with the RS1.2 or that Maven sent a cherry-picked unit the data is valid.

I was surprised to find Maven in bottom 5 but I guess whatever they’re doing with the RS1.2 hasn’t translated across all their lines yet.

Realistically Maven making a solid scope in the RS1.2 but struggling with drop test durability overall isn’t any different than Leupold making a bunch of optics that perform poorly while the 10x Ultra M3A appears solid.

 

[Formidilosus]

You should only remove outliers if you suspect that it is from poor testing or bad sampling. Unless you think Form didn’t do the test the same way with the RS1.2 or that Maven sent a cherry-picked unit the data is valid.

I was surprised to find Maven in bottom 5 but I guess whatever they’re doing with the RS1.2 hasn’t translated across all their lines yet.

Realistically Maven making a solid scope in the RS1.2 but struggling with drop test durability overall isn’t any different than Leupold making a bunch of optics that perform poorly while the 10x Ultra M3A appears solid.

For most of the companies- especially ones getting their scopes made by OEM’s, you tend to need to view each one on its own and not as a “brand”. Because the brand is mostly just a label having little to do with the actual scope in most instances.

 

[chanson_roland]

PDF version

Please post the "subscribe" link for your podcast!



Thanks for doing this. The data don't lie.

 

[Bado20]

For most of the companies- especially ones getting their scopes made by OEM’s, you tend to need to view each one on its own and not as a “brand”. Because the brand is mostly just a label having little to do with the actual scope in most instances.

This may be true for most companies but it is clearly not true for all of them, the results show that at least SWFA and Trijicon, and likely NF as well are doing something across their lines that is causing them to do better on these drop tests.

As far as comparing a specific model, it would be a useful comparison but there would need to be test data for at least 4 examples of that model to have a chance at any meaningful inference.

 

[sndmn11]

You should only remove outliers if you suspect that it is from poor testing or bad sampling. Unless you think Form didn’t do the test the same way with the RS1.2 or that Maven sent a cherry-picked unit the data is valid.

I was surprised to find Maven in bottom 5 but I guess whatever they’re doing with the RS1.2 hasn’t translated across all their lines yet.

Realistically Maven making a solid scope in the RS1.2 but struggling with drop test durability overall isn’t any different than Leupold making a bunch of optics that perform poorly while the 10x Ultra M3A appears solid.

There's strong correlation in what OEM is actually producing the optic, than the brand stamped on it. This should apply to the RS1.2, RS6, Bushnell LRHS/LRTS/LRHS2, NF, etc. The car battery analogy applies to good rifle scopes.

Like this guy says:

For most of the companies- especially ones getting their scopes made by OEM’s, you tend to need to view each one on its own and not as a “brand”. Because the brand is mostly just a label having little to do with the actual scope in most instances.

 

[mxgsfmdpx]

This may be true for most companies but it is clearly not true for all of them, the results show that at least SWFA and Trijicon, and likely NF as well are doing something across their lines that is causing them to do better on these drop tests.

As far as comparing a specific model, it would be a useful comparison but there would need to be test data for at least 4 examples of that model to have a chance at any meaningful inference.

No.

In general. You cannot base your data dump on “brands”. They don’t make anything and saying this “brand” scope is or is not reliable is not the way.

It must be model specific within the design houses.

 

[omicron1792]

No.

In general. You cannot base your data dump on “brands”. They don’t make anything and saying this “brand” scope is or is not reliable is not the way.

It must be model specific within the design houses.

I think he gets that point.

His point is you can’t get statistical significance with one scope here and one scope there.

 

[Bado20]

No.

In general. You cannot base your data dump on “brands”. They don’t make anything and saying this “brand” scope is or is not reliable is not the way.

It must be model specific within the design houses.

I understand that looking at specific models would be ideal, at this point there isn't enough data to do that.

Also, some of the "brands" do perform better. I don't think any of us are stupid enough to think it is the logo on the turret that is causing them to do so. Obviously brand is a proxy for something else, be it a design team, OEM, whatever. But you don't get a 0.0003 p-value on something by random chance.

 

[cuerro viejo]

I understand that looking at specific models would be ideal, at this point there isn't enough data to do that.

Also, some of the "brands" do perform better. I don't think any of us are stupid enough to think it is the logo on the turret that is causing them to do so. Obviously brand is a proxy for something else, be it a design team, OEM, whatever. But you don't get a 0.0003 p-value on something by random chance.

at what point do you just use the actual manufacturer facility location and not brand then? There is already I high degree of correlation there…

How many different factories/mfg does vortex use?

 

[Bado20]

at what point do you just use the actual manufacturer facility location and not brand then? There is already I high degree of correlation there…

How many different factories/mfg does vortex use?

It would be interesting to look at. I don’t know where to find the manufacturing plants for all the specific models of scope though.

 

[Formidilosus]

This may be true for most companies but it is clearly not true for all of them, the results show that at least SWFA and Trijicon, and likely NF as well are doing something across their lines that is causing them to do better on these drop tests.

That is true. The difference is this three all openly prioritize durability and reliability. Vortex uses the same OEM for some of their scopes, yet because of their choices in specifics they do not tend to be durable or reliable.

If just looking at broad views of it- SWFA, Trijicon and NF are clearly offer the highest probability of good service from their scopes. But, there are very good models from others as well. And when looking at those, you have to view them as a stand alone option- the Leupold Mark 4 fixed powers as an example.

As far as comparing a specific model, it would be a useful comparison but there would need to be test data for at least 4 examples of that model to have a chance at any meaningful inference.

Kind of. This is probably too deep for me to type on my phone- but think total failure of designed intent. If using a truck for instance you do not need to crash 30 of them (or even 4) if the first 1 or 2 has the motor getting pushed into the back seat from a front end 15mph impact. The design at that point is clearly and unequivocally too fragile to be safe for use.

However, to see the actual difference between two good options say between a Leupold fixed power Mark 4 and a Trijicon Tenmile- yes, lots of samples need to be tested, and rigorously so. BUT that isn’t what the initial eval is looking for- it’s functionally looking for which cars (scopes) are so unsafe (loss of zero) that they shouldn’t even be considered.

 

[Bado20]

If just looking at broad views of it- SWFA, Trijicon and NF are clearly offer the highest probability of good service from their scopes. But, there are very good models from others as well. And when looking at those, you have to view them as a stand alone option- the Leupold Mark 4 fixed powers as an example.

I'm not trying to discount that there are good scopes that are pretty much a standalone option compared to the rest of the companies offerings. Where this is more relevant is if someone is looking to buy a scope that has not been drop tested already. If going off of the drop test list you have a higher probability of durability going with a SWFA than a Vortex.

Kind of. This is probably too deep for me to type on my phone- but think total failure of designed intent. If using a truck for instance you do not need to crash 30 of them (or even 4) if the first 1 or 2 has the motor getting pushed into the back seat from a front end 15mph impact. The design at that point is clearly and unequivocally too fragile to be safe for use.

However, to see the actual difference between two good options say between a Leupold fixed power Mark 4 and a Trijicon Tenmile- yes, lots of samples need to be tested, and rigorously so. BUT that isn’t what the initial eval is looking for- it’s functionally looking for which cars (scopes) are so unsafe (loss of zero) that they shouldn’t even be considered.

Yes, to prove something is durable is a lot harder than to prove it isn't.

It's similar to testing the accuracy of a rifle. If you are going out to test if your rifle fires a 1.5MOA cone and you shoot a 3 round group and it is 0.5 MOA you need to keep shooting because your sample size is too small. On the other hand if you go out and shoot a 2.5 MOA group with your first 3 shots you can save the ammo and stop there.

 

[Str8shooter]

Some stats in this world are sorely needed! I really want to buy some of these conclusions and I think the proportion of 0% misses vs high % misses conveys important statistical information. I do think other scope factors like tube diameter, price, weight are also worth assessing... However for this work I do wonder if the assumptions of normality are true here. Your shapiro-wilk test looks like it was done for the entire group but I think welches two sample test applies to each comparison group separately, not just to the pooled data. "Student's t-test assumes that the sample means being compared for two populations are normally distributed, and that the populations have equal variances. Welch's t-test is designed for unequal population variances, but the assumption of normality is maintained " Also shapiro-wilk is somewhat notorious for giving low (ie <.05 p-values) for small sample sizes which you have for each brand group.... I also think that the hit/miss construct makes for a binomial dataset that lends itself to other stats assessments like generalized linear modeling or something cool like permutation tests that R can run pretty easily check out the R coin package, giggity.

 

[Southern Lights]

I think the main advantage NF has is that they are beating on their scopes before being shipped and testing for function and shift. This is going to weed out many failures that end users would normally see.

It's akin to running a computer burn in for a few days. It eliminates a lot of failures very early on. For NF I suspect the duds just don't make it out the door.

Likewise, Trijicon built their rep on extremely tough military optics. I think they value that reputation to not ship fragile gear.

I would not be surprised at all to find those two vendors scoring consistently well in these evals. Their product brand is built on these ideas and it's what they market to.

 

[Formidilosus]

I think the main advantage NF has is that they are beating on their scopes before being shipped and testing for function and shift. This is going to weed out many failures that end users would normally see.

It's akin to running a computer burn in for a few days. It eliminates a lot of failures very early on. For NF I suspect the duds just don't make it out the door.

Likewise, Trijicon built their rep on extremely tough military optics. I think they value that reputation to not ship fragile gear.

I would not be surprised at all to find those two vendors scoring consistently well in these evals. Their product brand is built on these ideas and it's what they market to.

For Trijicon specially, their scopes were generally decent historically- however they did have issues at times with side impacts and high round counts. Around 2019 or so they started addressing it, and 2021 IIRC they renamed most of their scopes and started openly stating/advertising that their scopes were tested for zero retention from drops, 1,000/3,000 rounds, etc.