Hunting for failure in literature reviews

Methodology

Most of us default to searching for the win when doing a literature review. We put in our key terms and look for the papers that confirm the hypothesis. The problem is that the most interesting boundaries of a phenomenon are usually buried in the papers that didn't make the press release. Instead of searching for "X increases Y," try searching for "X" combined with negative strings. In PubMed or Google Scholar, add phrases like "did not find," "contrary to," or "failed to replicate" to your query. For example, if you are looking at a specific protein's effect on cell signaling, searching for "Protein X + 'no significant difference'" or "Protein X + 'failed to replicate'" often surfaces the studies that define exactly where the effect stops working. These are the papers that usually get skipped because they aren't breakthroughs, but they are the ones that actually prevent you from overgeneralizing your results. It flips the process from confirming a trend to mapping the edges of the data.

4 comments

Comments

MemoryHoleMarcus·1 hour ago

Searching for "failed to replicate" assumes those papers actually made it past the editor. The file drawer effect usually swallows those results long before they hit PubMed.

LurkingLorraine·1 hour ago

registered reports solve the file drawer problem by committing to publish regardless of outcome.

QuietOptimistQi·1 hour ago

This approach feels more viable now that pre-print servers are standard. We are seeing more null results stay visible before the traditional peer review filter removes them.

CuriousMarie·1 hour ago

but how do we tell if the failure was a real boundary of the phenomenon... or just a bad experimental design? that seems like the hardest part of the map...