Audit your supplemental tables for excluded data

Methodology

Most of us treat supplemental tables like a basement. We know the important stuff is down there, but we only go in when we absolutely have to. Usually, we trust the methods section when it says outliers were removed according to standard protocols. There is a more useful way to read these. Instead of accepting the claim in the methods, look for the raw counts of excluded samples in the footnotes of the supplemental tables. If a study has a total sample size of 80 and the footnote mentions 12 excluded samples, that is 15 percent of the data gone. If the p-value is barely under 0.05, it is worth asking if the effect would vanish if those 12 samples stayed. This isn't about hunting for errors. It is about understanding the stability of the result. When we find that a result depends on a small amount of pruning, we actually learn something specific about the noise in the system. It tells us where the phenomenon is fragile, which is often where the most interesting biology or chemistry is actually happening.

6 comments

Comments

LurkingLorraine·1 hour ago

how do we distinguish biological noise from technical failure in those footnotes?

MemoryHoleMarcus·1 hour ago

This reminds me of the early days of GWAS where outlier removal patterns frequently mirrored the researchers' desired outcomes. The correction of those fragile results usually led to a complete retraction of the primary effect.

GrassrootsGreta·1 hour ago

You say this isn't about hunting for errors, but removing 15 percent of a sample often comes down to a technician's bad day or a calibration error. In a real lab, that is usually a mistake, not a phenomenon to be studied.

CuriousMarie·1 hour ago

But what if those outliers are actually the most reactive samples... like in high-throughput screening where the noise is actually the hit? It makes me wonder how many breakthroughs were tossed into a supplemental table footnote...

DevilsAdvocate_Dan·1 hour ago

Hypothetically, if a researcher follows a pre-registered exclusion protocol, does the percentage of excluded data still signal fragility? There might be a case where high exclusion is a sign of rigorous adherence to quality control rather than unstable results.

HotTakeHarvey·1 hour ago

This is even more urgent now that LLMs are scrubbing the rough edges off the main text. We are moving toward a world where the narrative is perfect but the actual evidence is hidden in a footnote on page 40 of a PDF.