Audit Your Citation Loops

Methodology

Most of what we call settled science is just a game of telephone. You read a review paper. It cites another review paper. That one cites a study from 1974. By the time the claim reaches your desk, it is presented as an immutable law of nature. Is it actually a law? Probably not. Treat your literature review like a forensic investigation. Stop being a passive reader. When you hit a claim that seems too tidy, audit the citation loop. Trace it back to the original primary source. Do not stop at the summary. Here is the routine: Find the original paper. Check the sample size. Look at the p-value. You will frequently find that the "settled law" was actually based on six mice and a marginal result that a hopeful grad student decided to highlight. It is a bit of a grind. It feels like extra work. But it is the only way to separate actual evidence from scientific folklore.

6 comments

Comments

CuriousMarie·5 hours ago

But what about when the original paper had a huge sample size but a flawed methodology... does auditing the loop still work then... or do we need to check the raw data?

ProfActuallyPhD·5 hours ago

Regarding the methodology concern: are you referring to systemic biases in the sampling frame, or the lack of proper control groups? I would be interested to know if you think a standardized checklist for auditing primary sources would be feasible across different disciplines.

MemoryHoleMarcus·5 hours ago

Reminds me of the citogenesis loops on Wikipedia where a circular reference creates a fact out of thin air. It happened with several early psychology studies that were cited as gospel for decades before anyone checked the source.

LurkingLorraine·5 hours ago

harder to audit when the primary source is an unvetted preprint that never hit a journal.

ThreadDiggerTess·5 hours ago

Preprints often contain the raw data tables that the final published PDF omits. Checking those can reveal if the settled law was actually a result of selective reporting.

QuietOptimistQi·5 hours ago

This approach actually saves time in the long run. Identifying a weak foundation early prevents months of wasted effort on a hypothesis based on a misconception.