IDEAS home Printed from https://ideas.repec.org/p/diw/diwwpp/dp1572.html
   My bibliography  Save this paper

Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful

Author

Listed:
  • Benedikt Fecher
  • Mathis Fräßdorf
  • Gert G. Wagner

Abstract

We live in a time of increasing publication rates and specialization of scientific disciplines. More and more, the research community is facing the challenge of assuring the quality of research and maintaining trust in the scientific enterprise. Replication studies are necessary to detect erroneous research. Thus, the replicability of research is considered a hallmark of good scientific practice and it has lately become a key concern for research communities and science policy makers alike. In this case study we analyze perceptions and practices regarding replication studies in the social and behavioral sciences. Our analyses are based on a survey of almost 300 researchers that use data from the German Socio-Economic Panel Study (SOEP), a multidisciplinary longitudinal multicohort study. We find that more than two thirds of respondents disagree with the statement that replications are not worthwhile, because major mistakes will be found at some point anyway. Nevertheless, most respondents are not willing to spend their time to conduct replication studies. This situation can be characterized as a “tragedy of the commons”: everybody knows that replications are useful, but almost everybody counts on others to conduct them. Our most important finding concerning practical consequences is that among the few replications that are reported, a large majority is conducted in the context of teaching. In our view, this is a promising detail: in order to foster replicability, one avenue may be to make replication studies a mandatory part of curricula as well as of doctoral theses. Furthermore, we argue that replication studies need to be more attractive for researchers. For example, successful replications could be listed in the publication lists of replicated authors. Vice versa, data sharing needs to receive more recognition, for example by considering data production and subsequent data sharing as scientific output.

Suggested Citation

  • Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    • Handle: RePEc:diw:diwwpp:dp1572
    as

    Download full text from publisher

    File URL: https://www.diw.de/documents/publikationen/73/diw_01.c.531685.de/dp1572.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Carmen M. Reinhart & Kenneth S. Rogoff, 2010. "Growth in a Time of Debt," American Economic Review, American Economic Association, vol. 100(2), pages 573-578, May.
    2. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    3. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    4. Nicola Lacetera & Lorenzo Zirulia, 2011. "The Economics of Scientific Misconduct," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 27(3), pages 568-603.
    5. Isabella Peters & Peter Kraker & Elisabeth Lex & Christian Gumpenberger & Juan Gorraiz, 2016. "Research data explored: an extended analysis of citations and altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 723-744, May.
    6. Gert G. Wagner & Joachim R. Frick & Jürgen Schupp, 2007. "The German Socio-Economic Panel Study (SOEP) – Scope, Evolution and Enhancements," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 127(1), pages 139-169.
    7. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    2. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    3. Fecher, Benedikt & Fräßdorf, Mathis & Hebing, Marcel & Wagner, Gert G., 2017. "Replikationen, Reputation und gute wissenschaftliche Praxis," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 68(2-3), pages 154-158.
    4. Fels, Katja M., 2021. "Who nudges whom? Field experiments with public partners," Ruhr Economic Papers 906, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Gert G. Wagner, 2019. "Eine Ethik der Politikberatung gehört zur Forschungsethik," RatSWD Working Papers 269, German Data Forum (RatSWD).
    6. Peter Harremoës, 2019. "Replication Papers," Publications, MDPI, vol. 7(3), pages 1-8, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    2. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    3. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    4. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    5. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    6. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.
    7. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    8. Stefan Bach & Martin Beznoska & Viktor Steiner, 2011. "A Wealth Tax on the Rich to Bring down Public Debt?: Revenue and Distributional Effects of a Capital Levy," SOEPpapers on Multidisciplinary Panel Data Research 397, DIW Berlin, The German Socio-Economic Panel (SOEP).
    9. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    10. Kiri, Bralind & Lacetera, Nicola & Zirulia, Lorenzo, 2018. "Above a swamp: A theory of high-quality scientific production," Research Policy, Elsevier, vol. 47(5), pages 827-839.
    11. Vlaeminck, Sven & Herrmann, Lisa-Kristin, 2015. "Data Policies and Data Archives: A New Paradigm for Academic Publishing in Economic Sciences?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 145-155.
    12. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    13. W. Robert Reed, 2018. "A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 51(2), pages 286-300, June.
    14. Tom Coupé & W. Robert Reed & Christian Zimmerman, 2021. "Paving the Road for Replications: Experimental Results from an Online Research Repository," Working Papers in Economics 21/09, University of Canterbury, Department of Economics and Finance.
    15. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    16. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    17. Benedikt Fecher & Sascha Friesike & Marcel Hebing & Stephanie Linek & Armin Sauermann, 2015. "A Reputation Economy: Results from an Empirical Survey on Academic Data Sharing," RatSWD Working Papers 246, German Data Forum (RatSWD).
    18. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    19. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    20. Tom Coupé & W. Robert Reed & Christian Zimmermann, 2021. "Paving the Road for Replications: Experimental Results from an Online Research Bibliography," Working Papers 2021-013, Federal Reserve Bank of St. Louis, revised 24 Feb 2022.

    More about this item

    Keywords

    Replicability; good science; data sharing; research policy;
    All these keywords.

    JEL classification:

    • A13 - General Economics and Teaching - - General Economics - - - Relation of Economics to Social Values
    • A20 - General Economics and Teaching - - Economic Education and Teaching of Economics - - - General
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • D02 - Microeconomics - - General - - - Institutions: Design, Formation, Operations, and Impact
    • D23 - Microeconomics - - Production and Organizations - - - Organizational Behavior; Transaction Costs; Property Rights

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:diw:diwwpp:dp1572. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Bibliothek (email available below). General contact details of provider: https://edirc.repec.org/data/diwbede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.