IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/2gurz.html
   My bibliography  Save this paper

Deciding what to replicate: A formal definition of “replication value” and a decision model for replication study selection

Author

Listed:
  • Isager, Peder Mortvedt

    (Eindhoven University of Technology)

  • van Aert, Robbie Cornelis Maria
  • Bahník, Štěpán

    (University of Economics, Prague)

  • Brandt, Mark John

    (Tilburg University)

  • DeSoto, Kurt Andrew

    (Association for Psychological Science)

  • Giner-Sorolla, Roger
  • Krueger, Joachim
  • Perugini, Marco
  • Ropovik, Ivan

    (University of Presov)

  • van 't Veer, Anna Elisabeth

    (Leiden University)

Abstract

Robust scientific knowledge is contingent upon replication of original findings. However, researchers who conduct replication studies face a difficult problem; there are many more studies in need of replication than there are funds available for replicating. To select studies for replication efficiently, we need to understand which studies are the most in need of replication. In other words, we need to understand which replication efforts have the highest expected utility. In this article we propose a general rule for study selection in replication research based on the replication value of the claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by replicating the claim, and is a function of (1) the value of being certain about the claim, and (2) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value.

Suggested Citation

  • Isager, Peder Mortvedt & van Aert, Robbie Cornelis Maria & Bahník, Štěpán & Brandt, Mark John & DeSoto, Kurt Andrew & Giner-Sorolla, Roger & Krueger, Joachim & Perugini, Marco & Ropovik, Ivan & van 't, 2020. "Deciding what to replicate: A formal definition of “replication value” and a decision model for replication study selection," MetaArXiv 2gurz, Center for Open Science.
    • Handle: RePEc:osf:metaar:2gurz
    DOI: 10.31219/osf.io/2gurz
    as

    Download full text from publisher

    File URL: https://osf.io/download/5f4f4314a392b9002f1d9576/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/2gurz?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Purkayastha, Amrita & Palmaro, Eleonora & Falk-Krzesinski, Holly J. & Baas, Jeroen, 2019. "Comparison of two article-level, field-independent citation metrics: Field-Weighted Citation Impact (FWCI) and Relative Citation Ratio (RCR)," Journal of Informetrics, Elsevier, vol. 13(2), pages 635-642.
    2. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    3. Jörn Block & Andreas Kuckertz, 2018. "Seven principles of effective replication studies: strengthening the evidence base of management research," Management Review Quarterly, Springer, vol. 68(4), pages 355-359, November.
    4. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    5. Stephan Lewandowsky & Klaus Oberauer, 2020. "Low replicability can support robust and efficient science," Nature Communications, Nature, vol. 11(1), pages 1-12, December.
    6. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hussain, Walayat & Merigó, José M. & Rahimi, Iman & Lev, Benjamin, 2025. "Half a century of Omega – The International Journal of Management Science: A bibliometric analysis," Omega, Elsevier, vol. 133(C).
    2. Brinkerink, Jasper & De Massis, Alfredo & Kellermanns, Franz, 2022. "One finding is no finding: Toward a replication culture in family business research," Journal of Family Business Strategy, Elsevier, vol. 13(4).
    3. Jörn H. Block & Christian Fisch & Narmeen Kanwal & Solvej Lorenzen & Anna Schulze, 2023. "Replication studies in top management journals: An empirical investigation of prevalence, types, outcomes, and impact," Management Review Quarterly, Springer, vol. 73(3), pages 1109-1134, September.
    4. Nicolas Scelles & Jaime A. Teixeira da Silva, 2025. "Making the impact of publications within a field comparable by improving the field-weighted citation impact (FWCI): the case of sport management," Scientometrics, Springer;Akadémiai Kiadó, vol. 130(3), pages 1571-1586, March.
    5. Tian-Yuan Huang & Liying Yang, 2022. "Superior identification index: Quantifying the capability of academic journals to recognize good research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4023-4043, July.
    6. Lucy Semerjian & Kunle Okaiyeto & Mike O. Ojemaye & Temitope Cyrus Ekundayo & Aboi Igwaran & Anthony I. Okoh, 2021. "Global Systematic Mapping of Road Dust Research from 1906 to 2020: Research Gaps and Future Direction," Sustainability, MDPI, vol. 13(20), pages 1-21, October.
    7. Bornmann, Lutz & Haunschild, Robin, 2016. "Normalization of Mendeley reader impact on the reader- and paper-side: A comparison of the mean discipline normalized reader score (MDNRS) with the mean normalized reader score (MNRS) and bare reader ," Journal of Informetrics, Elsevier, vol. 10(3), pages 776-788.
    8. Prodhan Mahbub Ibna Seraj & Blanka Klimova & Rubina Khan, 2024. "Visualizing Research Trends in English Language Teaching (ELT) From 2013 to 2022: A Bibliometric Analysis," SAGE Open, , vol. 14(2), pages 21582440241, May.
    9. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    10. Alberto Saracco, 2022. "Dr. Strangelove: Or How I Learned to Stop Worrying and Love the Citations," The Mathematical Intelligencer, Springer, vol. 44(4), pages 326-330, December.
    11. Wenxuan Shi & Renli Wu, 2024. "Women’s strength in science: exploring the influence of female participation on research impact and innovation," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4529-4551, July.
    12. Manuel Marques-Cruz & Daniel Martinho Dias & João A. Fonseca & Bernardo Sousa-Pinto, 2024. "Ten year citation prediction model for systematic reviews using early years citation data," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4847-4862, August.
    13. Tom Coupé & W. Robert Reed, 2021. "Do Negative Replications Affect Citations?," Working Papers in Economics 21/14, University of Canterbury, Department of Economics and Finance.
    14. Guoqiang Liang & Ying Lou & Haiyan Hou, 2022. "Revisiting the disruptive index: evidence from the Nobel Prize-winning articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 5721-5730, October.
    15. Fuentes, Agustín & Espinoza, Ulises J. & Cobbs, Virginia, 2024. "Follow the citations: Tracing pathways of “race as biology” assumptions in medical algorithms in eGFR and spirometry," Social Science & Medicine, Elsevier, vol. 346(C).
    16. Bornmann, Lutz & Tekles, Alexander, 2021. "Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts," Journal of Informetrics, Elsevier, vol. 15(3).
    17. Zhichao Wang & Valentin Zelenyuk, 2021. "Performance Analysis of Hospitals in Australia and its Peers: A Systematic Review," CEPA Working Papers Series WP012021, School of Economics, University of Queensland, Australia.
    18. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    19. Hou, Jianhua & Yang, Xiucai, 2020. "Social media-based sleeping beauties: Defining, identifying and features," Journal of Informetrics, Elsevier, vol. 14(2).
    20. Scherp, Guido & Siegfried, Doreen & Biesenbender, Kristin & Breuer, Christian, 2020. "The role of Open Science in economics. Results report from an online survey among researchers in economics at German higher education institutions in 2019," EconStor Research Reports 222882, ZBW - Leibniz Information Centre for Economics.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:2gurz. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.