Stop using p-values to prove equivalence

Statistics

I keep seeing reports where someone runs a test, gets a p-value of 0.12, and concludes there is "no significant difference" between two options. That is a logic fail. Failing to find a difference isn't the same as proving things are equal; it usually just means the sample size was too small or the data was too noisy to see anything. If you actually need to prove that two things are practically the same, you need Equivalence Testing, specifically the Two One-Sided Tests (TOST) procedure. It moves the goalposts from "did I find something?" to "is the difference small enough to ignore?" Here is how to actually do it: First, define an equivalence margin. This is the practical limit where a difference stops mattering in the real world. If you are comparing two types of road sealant, maybe a 3% difference in durability is negligible for the budget. That 3% is your margin. Then, instead of one standard test, you run two separate one-sided tests: 1. Test if the difference is significantly greater than the lower bound of your margin. 2. Test if the difference is significantly less than the upper bound of your margin. If both tests are statistically significant, you have actually proven equivalence. You aren't just relying on a lack of evidence; you are proving the difference falls within a range that doesn't matter. It requires more legwork to define that margin upfront, but it's the only way to avoid the trap of claiming a non-result is a positive finding.

5 comments

Comments

SkepticalMike·1 hour ago

What is the standard for selecting that margin to ensure it isn't just tuned to fit the observed data?

MemoryHoleMarcus·1 hour ago

Claiming TOST is the only way is a stretch. Bayesian estimation of the effect size provides a distribution that addresses the 'absence of evidence' problem without needing a hard margin upfront.

CuriousMarie·1 hour ago

But the power calculations are so different... you can't just guess the sample size for a non-significant result and hope it means equivalence!

ProfActuallyPhD·1 hour ago

To build on the methodology, one should consider the 90% confidence interval. If the entire interval falls within the predefined equivalence margin, it is mathematically equivalent to passing both TOSTs.

LurkingLorraine·1 hour ago

especially problematic now that llms are synthesizing reviews and likely scanning for p < 0.05 without checking for tost.