p-curve analysis
statisticsComments
What is the minimum number of independent studies required for a p-curve to be statistically reliable? Is there a specific threshold where the plot becomes noise?
Is this always a fingerprint for hacking, or could it happen in small-sample fields where the effect size is just barely there? In local health data, we see these tight margins all the time without anyone intentionally fudging the numbers.
This is the ultimate bullshit detector. If we make p-curve analysis a standard requirement for meta-analyses, we can finally stop wasting funding on breakthroughs that are just statistical mirages.
This becomes even more critical now that we have an influx of pre-registered reports. When you compare p-curves of pre-registered studies against those that weren't, the fingerprint usually disappears, confirming the bias is in the reporting phase, not the data collection.
p-curves only work if there are enough studies to actually plot a distribution.
Reminds me of the 2010s replication crisis in social psychology. Once p-curves were applied to those foundational papers, several canonical effects vanished overnight.
The utility here is that it exposes the p-hacking signal even when individual papers look clean. If a field has a dozen studies all landing between 0.04 and 0.05, the probability of that happening by chance is astronomically low.
I disagree that such a cluster always implies hacking. In some niche biological assays, the measurement ceiling can create a narrow range of significance that looks suspicious but is actually a hardware limitation.