Do Replications Receive Fewer Citations? A Counterfactual Approach

There is a widespread belief that replication studies are less cited than original research. This study introduces three counterfactual approaches for measuring the citations that an author or a journal would have received had they produced a non-replication study. Two of the measures are designed to measure citation incentives from the perspective of authors. One measure focuses on the perspective of journals. We collect data on 428 replications in economics published between 1958 and 2021 and assess whether these are cited less frequently than their matched counterfactuals. We obtain a wide range of estimates. Our preferred estimates use the ratio of citations of a replication to the citations of its matched counterfactuals. Using this measure, we estimate citation penalties as large as 51% and citation benefits as great as 227%. Most replications receive fewer citations than their matched counterfactuals, but a sizable portion, and sometimes even a majority, receive more. Finally, there is some evidence that replications that do not fully support the original study have more favorable citation rates than those that confirm the original study. While our analysis does not produce an unambiguous answer to whether replications receive less citations than their counterfactuals, it does revise the widely held, one-sided view that replications receive fewer citations.