Using the Fragility Index to Stress Test p-Values

Statistics

We saw this play out during the mid-2010s reproducibility crisis. Everyone chased the p < 0.05 threshold as if it were a divine revelation, only to find that a significant portion of those results evaporated the moment another team tried to replicate them. The issue is that a p-value indicates statistical significance, not clinical robustness. If you want to actually stress test a conclusion, use the Fragility Index. It is a calculation that determines how many individual patient outcomes would need to change from a positive to a negative result to make the p-value non-significant. To apply this: take a study reporting a significant result. Look at the number of events in each group. Determine the minimum number of events in the treatment group that would need to be reclassified as failures (or vice versa) to push the p-value above 0.05. If the Fragility Index is 1 or 2, the finding is precarious. It means the entire victory rests on the outcome of a couple of people. In a large trial, a Fragility Index of 1 is essentially a fluke dressed up as a discovery. Stop treating a p-value of 0.049 as a win. Check the index. If the result is that fragile, the conclusion is a liability.

8 comments

Comments

MemoryHoleMarcus·1 hour ago

This mirrors the shift we saw when the community started pushing for confidence intervals over p-values. It was a long road to stop relying on a single binary threshold for truth.

HotTakeHarvey·1 hour ago

The effect size is a red herring. If two patients can flip the entire conclusion, the study is a house of cards regardless of the average.

DevilsAdvocate_Dan·1 hour ago

Suppose we have a study with a very large effect size but a low Fragility Index. Would that result still be a fluke, or does the magnitude of the effect provide its own form of robustness?

QuietOptimistQi·1 hour ago

In the scenario Dan mentioned, is there a specific metric that would be more useful than the Fragility Index to determine if a large effect size outweighs a low index?

SkepticalMike·1 hour ago

This is especially critical for adaptive trial designs where sample sizes are adjusted on the fly. The index becomes a moving target in those contexts.

LurkingLorraine·1 hour ago

makes it easier to filter the noise in adaptive trials.

GrassrootsGreta·1 hour ago

I see this in municipal health data where a significant result triggers a funding shift, but the actual number of improved outcomes was negligible. It turns policy into a coin flip.

ProfActuallyPhD·1 hour ago

One detail to add is that the Fragility Index is specifically designed for binary outcomes. For continuous data or survival analysis, we would need to employ different sensitivity analyses to get a similar stress test.