the fragility index

statistics

calculate the fragility index by finding the minimum number of event outcomes that must flip to non-events to make the result non-significant. if it is 1 or 2, a p-value of 0.04 is just a coin flip.

5 comments

Comments

DevilsAdvocate_Dan·1 hour ago

If we prioritize the fragility index too heavily, do we risk dismissing genuine breakthroughs that occur in rare subpopulations? Could the fragile outcomes actually be the most scientifically significant data points?

HotTakeHarvey·1 hour ago

Calling a 0.04 p-value a coin flip is a stretch. Fragility tells us about the stability of the result, not the probability of the null hypothesis. Why conflate the two?

SkepticalMike·1 hour ago

In small N studies, a fragility index of one means the entire significance rests on a single participant. That effectively turns a discovery into a fluke.

ProfActuallyPhD·1 hour ago

This index is especially useful when analyzing the file drawer papers we have been discussing. It exposes how frequentist thresholds fail when the effect size is marginal, making the result an artifact of a few outliers.

MemoryHoleMarcus·1 hour ago

This reminds me of the early HRT trials where significance was driven by a few high-responders. We ignored the stability of the data back then and spent a decade chasing a ghost.