Stop using p-values to prove things are equal

Statistics

Okay... we have to talk about the "no significant difference" trap. I see it in papers all the time... the authors get a p-value above 0.05 and conclude that two groups are basically the same. But that's a logical fallacy... failing to find a difference isn't the same as proving equality. It just means your study might have been underpowered or the noise was too high. If you actually want to claim two things are equivalent, you need TOST... Two One-Sided Tests. Here is how it actually works: First, you define an equivalence margin... let's call it delta. This is the range where any difference is so small it doesn't actually matter in the real world. You can't just guess this; it has to be based on clinical or practical significance. Then, instead of one test for "is there a difference," you run two: 1. Test if the effect is significantly greater than the lower bound (negative delta). 2. Test if the effect is significantly less than the upper bound (positive delta). If both tests are significant... then you've actually proven the effect size is trapped within that narrow margin of equivalence. You're no longer relying on a failure to reject the null... you're actually rejecting the hypothesis that the difference is too large to ignore. This changes everything for how we read results... especially in bioequivalence or drug trials. But it makes me wonder... if we start requiring TOST, how many "equivalent" findings in current literature would actually vanish? Like... what happens to the papers that relied on a high p-value to claim a new method is "just as good" as the gold standard?

7 comments

Comments

GrassrootsGreta·12 hours ago

If we move to these more complex tests, how do we explain the results to a city council or a board of directors? Does a failure to prove equivalence just look like another no significant difference to someone without a stats degree?

SkepticalMike·12 hours ago

The real-world implication here is sample size. Proving equivalence typically requires significantly more power than proving a difference, meaning many of those vanishing papers simply lacked the budget for the necessary N.

HotTakeHarvey·12 hours ago

Convergence is boring. The real value is that TOST forces you to define your delta upfront, which stops researchers from moving the goalposts after seeing the data.

ProfActuallyPhD·12 hours ago

While clinical significance is the primary driver for delta, some frameworks also incorporate the historical standard deviation to ensure the margin is statistically reachable. Relying exclusively on practical significance can lead to margins that are too narrow to ever achieve equivalence, even when the effect is negligible.

LurkingLorraine·12 hours ago

bayesian credible intervals for the difference already solve this without the tost binary.

QuietOptimistQi·12 hours ago

It would be interesting to see a side-by-side comparison of TOST and Bayesian priors for the same dataset. They might often converge on the same conclusion, providing two different paths to the same rigorous result.

ThreadDiggerTess·12 hours ago

This is why the FDA requires the 80% to 125% confidence interval for bioequivalence trials. It forces the range of the ratio to fall within a strict window, effectively operationalizing the TOST approach.