Using the Fragility Index to test p-value stability

Statistics

Most of us see a p-value under 0.05 and check the 'significant' box. The problem is that p-values are binary indicators. They do not tell you if your conclusion is resting on a house of cards. That is where the Fragility Index (FI) comes in. The Fragility Index is the smallest number of participants whose status would have to change from an event to a non-event (or vice versa) to make the p-value >= 0.05. It quantifies how robust a result actually is. Here is a concrete example: imagine a trial comparing a new drug to a placebo. The drug group has 15 events out of 100 patients. The placebo group has 25 events out of 100. The p-value is 0.04. If you change just two patients in the drug group from 'no event' to 'event', the p-value might climb to 0.06. In this case, the Fragility Index is 2. When the FI is low, the result is fragile. It means a tiny shift in a few individual outcomes would have completely changed the study's conclusion. This is a critical reality check for papers that barely scrape past the 0.05 threshold and claim a breakthrough. To calculate this, you can use a simple loop in R or Python. Start with your observed counts, increment the event count in the treatment group by one, and recalculate the p-value. Repeat this process until the result is no longer statistically significant. The number of iterations is your FI.

4 comments

Comments

CuriousMarie·2 hours ago

Does the loop method only work for binary outcomes... what happens if the data is continuous? That seems like it would change the logic of the Fragility Index entirely...

GrassrootsGreta·2 hours ago

In small community clinics, we often see these 'significant' results in tiny cohorts where one patient misreporting a symptom flips the whole conclusion. It makes these breakthrough claims feel like a gamble when we are the ones actually implementing the protocols.

ProfActuallyPhD·2 hours ago

That is precisely why the FI is essential for small-sample trials. It exposes when a result is driven by a few extreme outliers rather than a consistent treatment effect, flagging a high risk of Type I errors in underpowered studies.

SkepticalMike·2 hours ago

The FI is a useful heuristic, but it ignores the clinical significance of those few shifts. A fragile p-value is one problem; a result that is statistically significant but medically irrelevant is another.