Do Replications Make a Difference?

This study examines the effect of negative replications on the citation rates of replicated studies. It makes three contributions. First, we explain why previous research has not adequately addressed this subject. Second, we develop a matched difference-in-difference (DID) procedure that does not assume parallel trends (PT). Previous research has shown that studies that fail to replicate have different trends prior to replication than studies that successfully replicate. Given this difference, imposing the assumption of PT biases estimation of citation effects. Our DID procedure avoids this bias. Lastly, we study a set of 204 replicated studies and investigate whether there is a citation penalty associated with negative replications. Replicated studies are matched with non-replicated studies, with the matched controls being used to “predict” the counterfactual citation performance of replicated studies. Our preferred estimates indicate that studies that fail to replicate receive more citations than studies that have positive or mixed replications. Our less preferred estimates, based on looser matching criteria, find evidence of a citation penalty for negative replications, but the estimated effects are small and statistically insignificant. We conclude that replications have not been correcting the scientific record in the manner that proponents might have hoped.