Checking the OSF graveyard before trusting a breakthrough

Methodology

We have been here before. Remember the mid-2010s when every second groundbreaking social psychology study evaporated the moment someone else tried to run the same script? The outcome was a decade of wasted grant money and a lot of embarrassed retractions. High-impact journals still have a fetish for positive results; they want the breakthrough, not the report stating that the hypothesis was wrong five times in a row. This creates a prestige bias. If a paper claims a stunning effect size in a top-tier journal, treat it as a hypothesis rather than a fact until you check the Open Science Framework (OSF). The technique is straightforward. Go to OSF.io and search for the lead author or the study title. Look specifically for the pre-registration or the registered report. If the authors registered their analysis plan before collecting data, you can see if they pivoted their hypotheses halfway through to find a p-value under 0.05. If you find three other related projects by the same team that yielded null results but never made it to print, you are likely looking at a cherry-picked outlier. The OSF is not just a digital filing cabinet. It is where the actual science lives, while the journals provide the polished press release.

4 comments

Comments

HotTakeHarvey·2 hours ago

That is the classic excuse for HARKing. The replication crisis proved that confounding variables are often just convenient labels for a failed hypothesis.

SkepticalMike·2 hours ago

The critical distinction here is between simple pre-registration and registered reports. The latter removes the incentive to pivot because the peer review happens before data collection.

LurkingLorraine·2 hours ago

what about the labs that pre-register but never upload the final analysis code?

DevilsAdvocate_Dan·2 hours ago

Suppose a researcher discovers a confounding variable that makes the original pre-registration obsolete. In that scenario, wouldn't insisting on rigid adherence to the initial plan potentially stifle the discovery of a more accurate mechanism?