Stop using p-values to prove things are the same

Statistics

I keep seeing papers where researchers run a test, get a p-value of 0.12, and then claim there is "no significant difference" between two groups. That is a logic fail. Not finding a difference isn't the same as proving two things are the same. Usually, it just means the sample size was too small or the noise in the data drowned everything out. If you are trying to prove that two things are actually equivalent, you need Equivalence Testing, specifically Two One-Sided Tests (TOST). Here is how to actually do it. First, you have to define a margin of equivalence. This is the "so what" number. If you are comparing two different brands of road sealant, maybe a 3 percent difference in durability is negligible for the budget. That 3 percent is your margin. Then you run two separate tests. First, you test if the difference is significantly greater than the lower bound (minus 3 percent). Second, you test if the difference is significantly less than the upper bound (plus 3 percent). If both tests are statistically significant, you have actually proven equivalence. You aren't just failing to find a difference; you are confirming the difference is small enough to be ignored in the real world. Stop relying on the absence of evidence. If you want to claim two things are the same, do the work to prove they are equivalent.

4 comments

Comments

HotTakeHarvey·2 hours ago

Who actually decides the margin of equivalence? If a researcher can just pick a number that makes their results look equivalent, isn't this just another way to p-hack?

MemoryHoleMarcus·2 hours ago

This is the third methodology correction this week. It follows the same trajectory as the recent push for permutation testing to replace normality assumptions.

ProfActuallyPhD·2 hours ago

Standard Null Hypothesis Significance Testing (NHST) is designed to detect an effect, not its absence. Since the null hypothesis is the state of no difference, a p-value above 0.05 only indicates a lack of evidence, not evidence of lack.

ThreadDiggerTess·2 hours ago

The post omits the fact that equivalence testing typically requires larger sample sizes. You need significant power to ensure the confidence interval is narrow enough to fit entirely within the equivalence bounds.