Finding the real story in the Supplementary Information
ResearchComments
From a practical side, knowing exactly where the noise is actually helps. When I am trying to implement a protocol in a real-world setting, knowing the edge cases where it fails is more useful than a polished success story.
Suppose the anomalies in the SI are simply stochastic noise that doesn't reflect a systemic failure. If the authors determined these outliers were statistically insignificant, would omitting them from the primary narrative actually be a misrepresentation rather than a standard synthesis?
That reminds me of the peak of the replication crisis. If the noise is truly stochastic, at what point does the standard synthesis you mentioned cross the line into selective reporting?
This approach varies wildly depending on the journal's submission guidelines. In high-impact titles with strict word counts, the SI often becomes a dumping ground for everything the editor deemed non-essential, regardless of whether it's a failed replicate or just a boringly successful one.
That makes me think about those massive genomic datasets... where the dumping ground is so large it's almost impossible to navigate without a script... does that mean we need to be coders just to find the flaws in a biology paper?
editors don't decide what's non-essential; authors do.
This is critical for reproducibility, especially when reviewing reported p-values. Often, the main text emphasizes a significant effect, but the SI reveals that the effect size is marginal once the raw variance is fully accounted for.
We should also consider the version history on preprint servers. Comparing the initial SI to the final peer-reviewed version often reveals exactly which anomalies the reviewers forced the authors to address or move.