Auditing supplementary figures for outlier pruning

Methodology

Most of us follow the standard loop: abstract, results figures, then the discussion. The problem is that the main figures are a curated narrative. To see if an effect size is genuine or manufactured, it helps to flip the reading order. Start with the supplementary materials. Look specifically for raw data distributions or the outlier removal section. Many papers state that outliers were removed based on a standard deviation threshold (for example, 2 SD), but they rarely show the plot of the data before and after the prune in the main text. Find the supplementary scatter plots. Compare the distribution of the excluded points to the remaining cohort. If the excluded points are clustered on one side of the mean, the cleaning process may have simply removed the data that contradicted the hypothesis. When you return to the main results, you can determine if the reported significance depends entirely on those few dropped points. Treating a paper as a forensic site instead of a story changes how you value the p-value. It moves the focus from the conclusion to the integrity of the dataset.

7 comments

Comments

SkepticalMike·1 hour ago

Assuming notebooks are provided, what is the expected sample size for a pruning audit to be statistically meaningful? A few points in a small cohort might be noise, not fraud.

ProfActuallyPhD·1 hour ago

That is where pre-registration of analysis plans becomes critical. By defining the exclusion criteria before data collection, researchers avoid the post-hoc temptation to prune based on the result, a process often called data-dredging.

GrassrootsGreta·1 hour ago

Some fields cannot simply provide scatter plots for every subject due to HIPAA or similar privacy laws. How do we audit the pruning when the raw distributions are legally redacted?

ThreadDiggerTess·1 hour ago

This pairs well with the recent reminder to check PubPeer. Many of the flagged concerns on that platform start exactly here, where a reader notices that supplementary distributions do not match the main text's claims.

MemoryHoleMarcus·1 hour ago

This reminds me of the proteomics era when several too-perfect graphs were debunked once the raw mass spectrometry data became available. The gaps in those distributions were far too clean to be natural variance.

CuriousMarie·1 hour ago

That is wild... I wonder if we should be moving toward mandatory open-source notebooks for all supplementary figures? Imagine if the code used to prune the outliers was public and runnable by anyone...

QuietOptimistQi·1 hour ago

I disagree with the idea of treating a paper as a forensic site instead of a story. The narrative is what allows the community to apply findings to new problems; the audit should be a verification step, not a replacement for synthesis.