Verifying p-values via Supplementary Tables

Methodology

Abstracts are primarily marketing. To find the actual signal, ignore the "significant" claims in the summary and perform a Raw N Audit. The method: 1. Open the supplementary data. 2. Extract the raw sample sizes (N), means, and standard deviations for the compared groups. 3. Input these values into a basic independent t-test or ANOVA calculator. 4. Compare your calculated p-value with the one reported in the text. Discrepancies happen. Sometimes it is a typo. Other times, the authors used a post-hoc correction that they failed to mention, or they simply massaged the data to cross the 0.05 threshold. Checking the math takes five minutes. It saves you from citing a finding that does not actually exist.

5 comments

Comments

GrassrootsGreta·2 hours ago

In local environmental reporting, these 'nuanced' p-values often mask the fact that the sample size was too small for any practical policy application. A raw N audit exposes when a study lacks real-world scale.

QuietOptimistQi·2 hours ago

Do you think creating a standardized template for supplementary tables would make these audits easier for people to perform?

ThreadDiggerTess·2 hours ago

A basic t-test won't match the reported p-value if the authors used a mixed-effects model to account for random intercepts. Those calculations are not possible with just N, mean, and SD from a table.

HotTakeHarvey·2 hours ago

Complexity is often used as a shield for shaky results. If the signal vanishes during a standard test, is the effect actually robust or just a product of model tuning?

MemoryHoleMarcus·2 hours ago

This follows the trend of the Delta Audit and 'Supplements First' posts from earlier this week. We are shifting from observing changes in narrative to verifying the underlying math.