Questioning the "gold Standard Drop Test" and the conclusions of "This scope brand does/doesn't hold zero"

[Chris in TN]

I'm going to walk away, mull this over for a while, and attempt to come up with a well-reasoned but nuanced reply to the OP.

Just kidding. This is the internet. I'm just tagging in case people start dropping some good memes later.

The drop tests here are not perfect and never will be. But they are all that most of us have. I can't go buy every scope I want and test it then gamble on getting it fixed if I butcher it in the process. So, they aren't perfect, but they are the best thing going in the optics world and it's a crying shame there isn't a way to reward those who are involved in them. Hopefully OP grasps that he isn't the only person who's ever completed a stats class or walked through an experimental design.

We might as well salvage this trainwreck by discussing everyone's favorite elk-wallop round. Three-thirty-eight Winchester Mangle'um or stay home, right?

 

[tstowater]

As opposed to all the threads that stay on the rails so well on the hide or 24HCF?

Don't spend any time on either of those forums. I don't mind a little dumpster fire, but they can have a tendency to drown out the good information. Lots of good information on Rokslide. That's why I spend time here.

 

[Walker6]

From post #2 in the original field eval thread:







————-

Seems like a pretty low bar to clear, and the field evals let it be known that the scope that left your assembly line on an average Tuesday can’t clear this bar.

Small sample size? Yes, but if I eat at a new burger place and they serve me a dry patty on a cold bun with a can of Pepsi I ain’t goin back for another.

 

[Marbles]

question the interpretations of many around the forum as gospel.

I think you over estimate people's belief in the test to get a straw man.

Ordering a CBC on every trauma doesn't mean you need one resulted to transfuse blood. If by the luck of the draw an ED provider goes months without seeing such a case and always has an H&H prior to transfusion, that does not mean they think the H&H is gospel or don't understand its limitations.

If you do run into a someone who does (edit, should read does not) understand H&H in early hemorrhage, you hopefully educate that one person rather than demanding the whole department shows up for an in-service.

I’m not sure I understand what you mean by “snap back”, so I don’t appreciate any irony.

The irony is at this point you are the EMT trying to tell the surgeon they don't understand an H&H in trauma patients. The EMT may be correct, but probably isn't and more than just the surgeon is going to give the EMT the side eye. Which, I can see the similarities to a non-medical person spewing nonsense, even though you are NOT at that level, I appreciate the irony. But, life is also full of irony, including my own posts on this topic.

To put it differently, we have hashed out your criticisms more than a few times. They are valuable, but after the value has been extracted, continuing to press the husks starts to irritate those who have been turning the wheel for a while. At least throw new stock in the press.

And that’s the rub, with all of my education and experience, I realize that there is no statistical significance in either.

You could have just said the results aren't statistically significant and saved a lot of typing. However, you would be wrong. QC routinely rejects an entire run due to a single failure because the likelihood of that failure being a false rejection is low, making it statistically significant.

This is the blindness of dealing with statistics from one area and thinking you understand them in others. It is the hospitalist thinking they understand the OR or a surgeon choosing not to consult cardiology on the post op patient who develops florid HF.

 

[QuikFire]

What do you shoot primarily? (Optics wise) and why did you pick said brand?

Nothing impressive -
custom rem 700 7mm bought used with a Nikon Monarch - its decent glass considering it came essentially free on this rig that someone apparently couldn't get to shoot because the front action screw was too long for the HS precision stock so the bolt would bind, or, the stock was just loose and shot terribly. I ground that down and have a 1/2 moa gun that I paid $900 for complete.
Weatherby Vanguard .308 bought used with an Athlon Argos - it's not great
Put an Athlon Talos BTR on my AR for a cheap shooting range toy - just fine for what it is, but not great.
Just bought a 6.5 PRC Howa 1500 Carbon stalker - looking for an optic for it, hence why I'm here.


None of these I have an loyalty towards, think Athlon is good value for money.

All the facts in the world will not make a impact on sales , most "shooters" / weekend warriors will not do the research , they will buy what their friend has or what is on sale . Or buy a product for the "great" warranty .
And 90 percent of people shooting are not good enough to tell if their scope holds zero or not .
Facts don't matter in retail sales and only a few will take the time to use the data if it was published , so who would pay for it ?
In our capitalistic society cash is king , not facts. IMO

This is precisely what I want more of; accurate data so that you, I and all of us who like to go down the rabbit hole with gear selection have some (more) reliable info/data to guide decisions on what fits our application best. That's all. It's honestly part of the process I thoroughly enjoy. I don't expect the manufacturers to produce this and make it available, but it sure would be nice if they did. Thus, I appreciate Form for doing what he does.



I do a ton of tinkering with my archery shit. build my own arrows, geek out over FOC, speed/energy, build an arrow based on components rather than buying an OTC package... It's clearly part of the process of hunting that we are able to do out of our garage and at our desktop on the forums, etc. I'm just looking for some more tangible info.

 

[eric1115]

And that’s the rub, with all of my education and experience, I realize that there is no statistical significance in either.

Can you elaborate on this? My small brain can't get around the idea that 2/2 failures indicates either such poor production QC on a sound design, or a fundamental flaw in the design itself, that it's not a model I'm interested in trying. If 10% of them aren't built right, there's a 1% chance of failing 2/2. If 30% aren't built right, there's a 9% chance of failing 2/2.

If you switched scalpel manufacturers and the blade fell off the first two you used, would you say there's no statistical significance in that?

 

[rbutcher1234]

I think you over estimate people's belief in the test to get a straw man.

Ordering a CBC on every trauma doesn't mean you need one resulted to transfuse blood. If by the luck of the draw an ED provider goes months without seeing such a case and always has an H&H prior to transfusion, that does not mean they think the H&H is gospel or don't understand its limitations.

If you do run into a someone who does understand H&H in early hemorrhage, you hopefully educate that one person rather than demanding the whole department shows up for an in-service.


The irony is at this point you are the EMT trying to tell the surgeon they don't understand an H&H in trauma patients. The EMT may be correct, but probably isn't and more than just the surgeon is going to give the EMT the side eye. Which, I can see the similarities to a non-medical person spewing nonsense, even though you are NOT at that level, I appreciate the irony. But, life is also full of irony, including my own posts on this topic.

To put it differently, we have hashed out your criticisms more than a few times. They are valuable, but after the value has been extracted, continuing to press the husks starts to irritate those who have been turning the wheel for a while. At least throw new stock in the press. View attachment 1047466

You could have just said the results aren't statistically significant and saved a lot of typing. However, you would be wrong. QC routinely rejects an entire run due to a single failure because the likelihood of that failure being a false rejection is low, making it statistically significant.

This is the blindness of dealing with statistics from one area and thinking you understand them in others. It is the hospitalist thinking they understand the OR or a surgeon choosing not to consult cardiology on the post op patient who develops florid HF.

You saved me the trouble of typing my reply on statistical significance and false positive/false negatives. Thank you.

 

[AZDC]

I’m of the opinion that if you want to make something better than YOU should be the one who does it.

While I appreciate the “doctor” background, I’ve met some really “smart” people say and do some really stupid things and have also had some great ideas come from unsuspecting people.

There’s a ton of reviews on gear out there and it’s all the same. They’re watered down, just describe it visually and honestly tells you absolutely nothing. I appreciate the drop test. It’s a free data point to help me spend my money better, that’s why I got a Trijicon Scope and I love it. It’s a tank and holds zero. It’s another resource

 

[Hnthrdr]

Nothing impressive -
custom rem 700 7mm bought used with a Nikon Monarch - its decent glass considering it came essentially free on this rig that someone apparently couldn't get to shoot because the front action screw was too long for the HS precision stock so the bolt would bind, or, the stock was just loose and shot terribly. I ground that down and have a 1/2 moa gun that I paid $900 for complete.
Weatherby Vanguard .308 bought used with an Athlon Argos - it's not great
Put an Athlon Talos BTR on my AR for a cheap shooting range toy - just fine for what it is, but not great.
Just bought a 6.5 PRC Howa 1500 Carbon stalker - looking for an optic for it, hence why I'm here.


None of these I have an loyalty towards, think Athlon is good value for money.

This is precisely what I want more of; accurate data so that you, I and all of us who like to go down the rabbit hole with gear selection have some (more) reliable info/data to guide decisions on what fits our application best. That's all. It's honestly part of the process I thoroughly enjoy. I don't expect the manufacturers to produce this and make it available, but it sure would be nice if they did. Thus, I appreciate Form for doing what he does.



I do a ton of tinkering with my archery shit. build my own arrows, geek out over FOC, speed/energy, build an arrow based on components rather than buying an OTC package... It's clearly part of the process of hunting that we are able to do out of our garage and at our desktop on the forums, etc. I'm just looking for some more tangible info.

Gotcha, you seem like many of us who stumbled into this site looking for answers.

Is the drop test gospel. No

Is it the closest thing that to my knowledge has been attempted, to find companies who build an aiming device that does what it’s supposed to do which is give the shooter an aim point that is mostly solid and repeatable?
Yes.

I used the tests to get me going in the right direction but ultimately it’s on me to confirm that my optics are mounted and zeroed correctly and are up to the task for my style of hunting.

I look at the drop tests as a guideline or a recipe and then I go from there to make my own choices, but to think that the findings it offers are incomplete because it is not perfect by the scientific standards, well now that’s throwing the baby out with the bath water…

 

[QuikFire]

I think you over estimate people's belief in the test to get a straw man.

Ordering a CBC on every trauma doesn't mean you need one resulted to transfuse blood. If by the luck of the draw an ED provider goes months without seeing such a case and always has an H&H prior to transfusion, that does not mean they think the H&H is gospel or don't understand its limitations.

If you do run into a someone who does understand H&H in early hemorrhage, you hopefully educate that one person rather than demanding the whole department shows up for an in-service.


The irony is at this point you are the EMT trying to tell the surgeon they don't understand an H&H in trauma patients. The EMT may be correct, but probably isn't and more than just the surgeon is going to give the EMT the side eye. Which, I can see the similarities to a non-medical person spewing nonsense, even though you are NOT at that level, I appreciate the irony. But, life is also full of irony, including my own posts on this topic.

To put it differently, we have hashed out your criticisms more than a few times. They are valuable, but after the value has been extracted, continuing to press the husks starts to irritate those who have been turning the wheel for a while. At least throw new stock in the press. View attachment 1047466

You could have just said the results aren't statistically significant and saved a lot of typing. However, you would be wrong. QC routinely rejects an entire run due to a single failure because the likelihood of that failure being a false rejection is low, making it statistically significant.

This is the blindness of dealing with statistics from one area and thinking you understand them in others. It is the hospitalist thinking they understand the OR or a surgeon choosing not to consult cardiology on the post op patient who develops florid HF.

1. Initial resusc has nothing to do with labs, so of course, H&H is not used as gospel at all, rather just a piece of evidence in more importantly, non hemorrhagic settings or to establish a trend. most often its resulted (in trauma) after action/transfusion has commenced. I don't infer everyone believes this is gospel, some do, and some adamantly defend it. We have collected enough data and published good studies to give an inclination on when to tranfuse in other scenarios - all the while understanding the limitations of that study, the population, etc.
2. I didn't post this as any sort of demand that anyone show up, I just wrote out my thoughts and gave you, me, and everyone else following the opportunity for engagement, possible entertainment, and a discussion that some find interesting enough to hang around.
2.75: I see, sorry I'm late to the party...
3. I follow and appreciate your analogies.
4. So positive predictive value of failure - yes, agree, obvious - but we didn't get there from an N of one - with a test prone to error.

 

[QuikFire]

Can you elaborate on this? My small brain can't get around the idea that 2/2 failures indicates either such poor production QC on a sound design, or a fundamental flaw in the design itself, that it's not a model I'm interested in trying. If 10% of them aren't built right, there's a 1% chance of failing 2/2. If 30% aren't built right, there's a 9% chance of failing 2/2.

If you switched scalpel manufacturers and the blade fell off the first two you used, would you say there's no statistical significance in that?

In the OP, I state the test does have a 100% true positive rate to detect failure; we agree there from the get go. But whats the false negative rate? Is the N of one representative of the optic line? That is my concern with the - not statistical significance - but statistical validity. If we don't have the latter, we can't know the former.

 

[AKborn87]

As others have said, it's not a perfect test, and maybe it's already been brought up, but it seems like two obvious ways to standardize the test could be to 1) use the same set of "bombproof" rings (whichever brand that is for both 1" and 30mm scopes), and 2) use the same short and/or long action rifle. Then at least the only thing changing among tests is the scope...

 

[QuikFire]

We might as well salvage this trainwreck by discussing everyone's favorite elk-wallop round. Three-thirty-eight Winchester Mangle'um or stay home, right?

475 gr broadhead at 295 FPS lol

really though, 7mm rem mag as far as I have any experience with thus far. Very keen on the 7mm backcountry.

 

[QuikFire]

As others have said, it's not a perfect test, and maybe it's already been brought up, but it seems like two obvious ways to standardize the test could be to 1) use the same set of "bombproof" rings (whichever brand that is for both 1" and 30mm scopes), and 2) use the same short and/or long action rifle. Then at least the only thing changing among tests is the scope...

Agree, agree.
I see (now much more than before) the field 'validity' of the drop test. Though internally I grind with the point of impact variation; ie the luck that a strike was positioned to cause variance in zero vs another that didn't. IDK, overthinking, but that's what I'm here for.

 

[Marbles]

Perhaps a better way to put it, the drop test is equivalent to a phase 1 clinical trial.

The ops criticism is equivalent to saying a drug that failed phase 1 trails wasn't given a fair chance and should go through phase 3 trails before being rejected.

The worst part is manufacturers of optics will not even pay for phase 1 trails and we have to. Then people insist we should also pay for phase 3 trials after the product fails phase 1.

 

[QuikFire]

I look at the drop tests as a guideline or a recipe and then I go from there to make my own choices, but to think that the findings it offers are incomplete because it is not perfect by the scientific standards, well now that’s throwing the baby out with the bath water…

Very well said.

 

[MTlan66]

Wow. I spent last night sleeping, and this morning shooting. I had no idea what I was missing!

 

[QuikFire]

Perhaps a better way to put it, the drop test is equivalent to a phase 1 clinical trial.

The ops criticism is equivalent to saying a drug that failed phase 1 trails wasn't given a fair chance and should go through phase 3 trails before being rejected.

The worst part is manufacturers of optics will not even pay for phase 1 trails and we have to. Then people insist we should also pay for phase 3 trials after the product fails phase 1.

1. On the contrary... that's not what I said nor am saying... I make clear the failures of the drop test are perfectly accurate for that optic; making broader generalizations from the results (positive or negative tests) requires making a lot of assumptions. To liken this to a clinical drug trial - not sure why we would make that analogy, seems a bit of a deviation from what it is and I struggle to follow the comparison (other than that it too is a way of testing something). Anyhow, I'll engage it as repeat that I'm merely suggesting the methods of this "phase 1 test" COULD be augmented to perhaps remove error.

2. I agree.
We do pay for all the trials (in drugs).

 

[nm.otter]

people generalized results from X scope to blanket the brand from an N of one.

Not really the case here. There are single models which have passed the test (e.g. Maven RS 1.2) while other models from the same brand (at least the ones that have been tested) did not. That information is well known here and regularly repeated in discussion threads.

There are also brands with multiple scopes tested over multiple years and generations of product, to the point that those brands are on a kind of GRAS list here. Failures will be posted and discussed if users report them.

In the OP, I state the test does have a 100% true positive rate to detect failure; we agree there from the get go.

Perhaps worth noting that there is typically more than one failure. Occasionally from a formal re-test, but most commonly from user-supplied anecdotes.

But whats the false negative rate? Is the N of one representative of the optic line? That is my concern with the - not statistical significance - but statistical validity. If we don't have the latter, we can't know the former.

That's a much more difficult problem -- and something the manufacturers should be doing. The good news here is that these drop tests are becoming sufficiently well-known that manufacturers are starting to make changes both to their own testing and to OEM specifications.

 

[QuikFire]

That's a much more difficult problem -- and something the manufacturers should be doing. The good news here is that these drop tests are becoming sufficiently well-known that manufacturers are starting to make changes both to their own testing and to OEM specifications.

Bingo. So let's beat the shit out of our optics with a specified protocol - consolidate and publish our data, then send it back to the manufacturer for warranty. Then sell it, and buy the scopes that hold up to the highest impact data that fits their application. Force the companies to start listening - or better - who gives a shit, we just support the ones who do.

Seriously.

(I know this is a long game strategy, and I know this is what we're at least getting a taste of from the work already done).