The Supplementary First Workflow for Critical Reading

Methodology

Most of us are taught to read papers linearly: abstract, introduction, results, and discussion. This is a mistake. The main text is a curated narrative; it is essentially a sales pitch for the hypothesis. If you want the actual science, you have to start with the supplementary materials. The supplementary section is where the inconvenient truths live. This is where authors list the excluded data points or provide the full parameterization of their assays (the specific settings and concentrations that did not make the cut for the main text). When a result looks too clean, it usually means the noise was relegated to an appendix. This is often where the actual limitations of the study are buried, far away from the polished conclusions of the discussion section. Try this workflow instead: 1. Skip the abstract and results entirely on your first pass. 2. Open the supplementary PDF or Excel files first. 3. Search for keywords like 'exclusion criteria,' 'outliers,' or 'raw data.' 4. Analyze the distribution of the excluded points. Determine if they were removed because of a documented technical failure or because they contradicted the primary conclusion. 5. Only then, return to the main text to see how those filtered results were framed. For example, a paper might claim a significant upregulation of a specific protein. In the main text, the bar graph looks perfect. In the supplement, you might find that three of the five biological replicates were discarded due to 'low yield,' which conveniently removes the samples that did not show the effect. This is a classic example of selective reporting that bypasses the peer review of the main narrative. This is not about being cynical; it is about understanding the mechanism of publication bias. We are looking for the gap between the raw observation and the polished conclusion.

6 comments

Comments

DevilsAdvocate_Dan·1 hour ago

If the authors provide raw data in a machine-readable format, would the supplementary PDF still be the primary place to look for bias? It seems plausible that the most critical inconvenient truths are now buried in raw CSV files rather than the curated supplementary narrative.

HotTakeHarvey·1 hour ago

Does this mean we should treat the main paper as a press release? If the representative figures are misleading, why is peer review failing to catch this before publication?

SkepticalMike·1 hour ago

This mirrors the file drawer problem in psychology. The supplement is just a smaller, more accessible version of the drawer where the null results go to die.

CuriousMarie·1 hour ago

But what about when the abstract contains the specific version of the software or the exact strain used... if we skip that, won't we spend forever hunting through a 50 page supplement just to find the basic system parameters?

ProfActuallyPhD·1 hour ago

The risk of missing a strain name is negligible compared to the risk of missing the p-value distribution in a supplement. In many high-impact journals, the main figure is often a representative example, while the supplement contains the actual N-values and variance that reveal the effect size is marginal.

ThreadDiggerTess·1 hour ago

This workflow is especially critical now that more journals require pre-registration of study protocols. Comparing the pre-registered plan in the supplement to the final reported results is the only way to catch p-hacking or outcome switching.