IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/22-02.html
   My bibliography  Save this paper

Do Negative Replications Affect Citations?

Author

Abstract

This study examines the effect of negative replications on the citation rates of replicated studies. We study a set of 204 replicated studies in economics and compare their citation performance with an initial sample of 112,000 potential controls taken from Scopus. From this initial pool, we match each replicated study with multiple controls based on having comparable citation histories. We have two main findings. First, studies that are replicated receive somewhat more citations than their matched control studies. Second, there is no evidence that studies that receive negative replications suffer a penalty in the form of fewer citations.

Suggested Citation

  • Tom Coupé & W. Robert Reed, 2022. "Do Negative Replications Affect Citations?," Working Papers in Economics 22/02, University of Canterbury, Department of Economics and Finance.
    • Handle: RePEc:cbt:econwp:22/02
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/2202.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Paul J. Devereux & Robert A. Hart, 2010. "Forced to be Rich? Returns to Compulsory Schooling in Britain," Economic Journal, Royal Economic Society, vol. 120(549), pages 1345-1364, December.
    2. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    3. Jodi Schneider & Di Ye & Alison M. Hill & Ashley S. Whitehorn, 2020. "Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2877-2913, December.
    4. Anton Nakov, 2010. "Jackknife instrumental variables estimation: replication and extension of angrist, imbens and krueger (1999)," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1063-1066.
    5. DeSimone, Jeff, 2007. "Fraternity membership and binge drinking," Journal of Health Economics, Elsevier, vol. 26(5), pages 950-967, September.
    6. Philip Oreopoulos, 2006. "Estimating Average and Local Average Treatment Effects of Education when Compulsory Schooling Laws Really Matter," American Economic Review, American Economic Association, vol. 96(1), pages 152-175, March.
    7. Angrist, J D & Imbens, G W & Krueger, A B, 1999. "Jackknife Instrumental Variables Estimation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 14(1), pages 57-67, Jan.-Feb..
    8. Badi H Baltagi, 2010. "Narrow Replication of Serlenga and Shin (2007) gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(3), pages 505-506.
    9. Hoenig J. M. & Heisey D. M., 2001. "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis," The American Statistician, American Statistical Association, vol. 55, pages 19-24, February.
    10. Pierre Azoulay & Toby Stuart & Yanbo Wang, 2014. "Matthew: Effect or Fable?," Management Science, INFORMS, vol. 60(1), pages 92-109, January.
    11. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    12. Judit Bar-Ilan & Gali Halevi, 2018. "Temporal characteristics of retracted articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1771-1783, September.
    13. Rees, Daniel I. & Sabia, Joseph J., 2010. "Body weight and smoking initiation: Evidence from Add Health," Journal of Health Economics, Elsevier, vol. 29(5), pages 774-777, September.
    14. Cawley, John & Markowitz, Sara & Tauras, John, 2004. "Lighting up and slimming down: the effects of body weight and cigarette prices on adolescent smoking initiation," Journal of Health Economics, Elsevier, vol. 23(2), pages 293-311, March.
    15. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    16. Yongcheol Shin & Laura Serlenga, 2007. "Gravity models of intra-EU trade: application of the CCEP-HT estimation in heterogeneous panels with unobserved common time-specific factors," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(2), pages 361-381.
    17. Amar Hamoudi, 2010. "Exploring the Causal Machinery behind Sex Ratios at Birth: Does Hepatitis B Play a Role?," Economic Development and Cultural Change, University of Chicago Press, vol. 59(1), pages 1-21, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tom Coupé & W. Robert Reed, 2023. "Do Replications Make a Difference?," Working Papers in Economics 23/10, University of Canterbury, Department of Economics and Finance.
    2. Coupé, Tom & Reed, W. Robert & Zimmermann, Christian, 2023. "Getting seen: Results from an online experiment to draw more attention to replications," Research Policy, Elsevier, vol. 52(8).
    3. L. Pieroni & L. Salmasi, 2015. "Does Cigarette Smoking Affect Body Weight? Causal Estimates from the Clean Indoor Air Law Discontinuity," Economica, London School of Economics and Political Science, vol. 82(328), pages 671-704, October.
    4. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    5. John Cawley & Davide Dragone & Stephanie Von Hinke Kessler Scholder, 2016. "The Demand for Cigarettes as Derived from the Demand for Weight Loss: A Theoretical and Empirical Investigation," Health Economics, John Wiley & Sons, Ltd., vol. 25(1), pages 8-23, January.
    6. Wehby, George L. & Courtemanche, Charles J., 2012. "The heterogeneity of the cigarette price effect on body mass index," Journal of Health Economics, Elsevier, vol. 31(5), pages 719-729.
    7. Cygan-Rehm, Kamila, 2018. "Is additional schooling worthless? Revising the zero returns to compulsory schooling in Germany," VfS Annual Conference 2018 (Freiburg, Breisgau): Digital Economy 181528, Verein für Socialpolitik / German Economic Association.
    8. Clémentine Garrouste & Mathilde Godard, 2016. "The lasting health impact of leaving school in a bad economy : Britons in the 1970s recession," Post-Print hal-01408637, HAL.
    9. Marion Schmidt, 2024. "Why do some retracted articles continue to get cited?," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(12), pages 7535-7563, December.
    10. Liwiński, Jacek, 2018. "The Impact of Compulsory Schooling on Earnings. Evidence from the 1999 Education Reform in Poland," GLO Discussion Paper Series 253, Global Labor Organization (GLO).
    11. Stephen Machin & Olivier Marie & Sunčica Vujić, 2011. "The Crime Reducing Effect of Education," Economic Journal, Royal Economic Society, vol. 121(552), pages 463-484, May.
    12. Jonneke Bolhaar & Sander Gerritsen & Sonny Kuijpers & Karen van der Wiel, 2019. "Experimenting with dropout prevention policies," CPB Discussion Paper 400, CPB Netherlands Bureau for Economic Policy Analysis.
    13. Robert A. HartBy & Mirko Moro & J. Elizabeth Roberts, 2017. "Who gained from the introduction of free universal secondary education in England and Wales?," Oxford Economic Papers, Oxford University Press, vol. 69(3), pages 707-733.
    14. Jodi Schneider & Di Ye & Alison M. Hill & Ashley S. Whitehorn, 2020. "Continued post-retraction citation of a fraudulent clinical trial report, 11 years after it was retracted for falsifying data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2877-2913, December.
    15. Asena Caner & Cahit Guven & Cagla Okten & Seyhun Orcan Sakalli, 2016. "Gender Roles and the Education Gender Gap in Turkey," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 129(3), pages 1231-1254, December.
    16. Gradstein, Mark & Justman, Moshe, 2009. "Understanding Compulsory Schooling Legislation: A Formal Model and Implications for Empirical Analysis," IZA Discussion Papers 4420, Institute of Labor Economics (IZA).
    17. Gonzalo Marco-Cuenca & José Antonio Salvador-Oliván & Rosario Arquero-Avilés, 2021. "Fraud in scientific publications in the European Union. An analysis through their retractions," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5143-5164, June.
    18. Seeun Jung, 2015. "Does education affect risk aversion? Evidence from the British education reform," Applied Economics, Taylor & Francis Journals, vol. 47(28), pages 2924-2938, June.
    19. Timo Hener & Tanya Wilson, 2018. "Marital Age Gaps and Educational Homogamy – Evidence from a Compulsory Schooling Reform in the UK," ifo Working Paper Series 256, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    20. Martin Fischer & Martin Karlsson & Therese Nilsson & Nina Schwarz, 2020. "The Long-Term Effects of Long Terms – Compulsory Schooling Reforms in Sweden," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 2776-2823.

    More about this item

    Keywords

    Replications; Citations; Matching; Meta-science; Self-correcting science;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:22/02. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.