IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/2gurz.html
   My bibliography  Save this paper

Deciding what to replicate: A formal definition of “replication value” and a decision model for replication study selection

Author

Listed:
  • Isager, Peder Mortvedt

    (Eindhoven University of Technology)

  • van Aert, Robbie Cornelis Maria
  • Bahník, Štěpán

    (University of Economics, Prague)

  • Brandt, Mark John

    (Tilburg University)

  • DeSoto, Kurt Andrew

    (Association for Psychological Science)

  • Giner-Sorolla, Roger
  • Krueger, Joachim
  • Perugini, Marco
  • Ropovik, Ivan

    (University of Presov)

  • van 't Veer, Anna Elisabeth

    (Leiden University)

Abstract

Robust scientific knowledge is contingent upon replication of original findings. However, researchers who conduct replication studies face a difficult problem; there are many more studies in need of replication than there are funds available for replicating. To select studies for replication efficiently, we need to understand which studies are the most in need of replication. In other words, we need to understand which replication efforts have the highest expected utility. In this article we propose a general rule for study selection in replication research based on the replication value of the claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by replicating the claim, and is a function of (1) the value of being certain about the claim, and (2) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value.

Suggested Citation

  • Isager, Peder Mortvedt & van Aert, Robbie Cornelis Maria & Bahník, Štěpán & Brandt, Mark John & DeSoto, Kurt Andrew & Giner-Sorolla, Roger & Krueger, Joachim & Perugini, Marco & Ropovik, Ivan & van 't, 2020. "Deciding what to replicate: A formal definition of “replication value” and a decision model for replication study selection," MetaArXiv 2gurz, Center for Open Science.
    • Handle: RePEc:osf:metaar:2gurz
    DOI: 10.31219/osf.io/2gurz
    as

    Download full text from publisher

    File URL: https://osf.io/download/5f4f4314a392b9002f1d9576/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/2gurz?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    2. Purkayastha, Amrita & Palmaro, Eleonora & Falk-Krzesinski, Holly J. & Baas, Jeroen, 2019. "Comparison of two article-level, field-independent citation metrics: Field-Weighted Citation Impact (FWCI) and Relative Citation Ratio (RCR)," Journal of Informetrics, Elsevier, vol. 13(2), pages 635-642.
    3. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    4. Jörn Block & Andreas Kuckertz, 2018. "Seven principles of effective replication studies: strengthening the evidence base of management research," Management Review Quarterly, Springer, vol. 68(4), pages 355-359, November.
    5. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    6. Stephan Lewandowsky & Klaus Oberauer, 2020. "Low replicability can support robust and efficient science," Nature Communications, Nature, vol. 11(1), pages 1-12, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brinkerink, Jasper & De Massis, Alfredo & Kellermanns, Franz, 2022. "One finding is no finding: Toward a replication culture in family business research," Journal of Family Business Strategy, Elsevier, vol. 13(4).
    2. Alberto Saracco, 2022. "Dr. Strangelove: Or How I Learned to Stop Worrying and Love the Citations," The Mathematical Intelligencer, Springer, vol. 44(4), pages 326-330, December.
    3. Zhichao Wang & Valentin Zelenyuk, 2021. "Performance Analysis of Hospitals in Australia and its Peers: A Systematic Review," CEPA Working Papers Series WP012021, School of Economics, University of Queensland, Australia.
    4. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    5. Yaxue Ma & Zhichao Ba & Yuxiang Zhao & Jin Mao & Gang Li, 2021. "Understanding and predicting the dissemination of scientific papers on social media: a two-step simultaneous equation modeling–artificial neural network approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7051-7085, August.
    6. Katarzyna Piwowar‐Sulej, 2021. "Core functions of Sustainable Human Resource Management. A hybrid literature review with the use of H‐Classics methodology," Sustainable Development, John Wiley & Sons, Ltd., vol. 29(4), pages 671-693, July.
    7. Monika Blišťanová & Peter Koščák & Michaela Tirpáková & Magdaléna Ondicová, 2023. "A Cross-Comparative Analysis of Transportation Safety Research," Sustainability, MDPI, vol. 15(9), pages 1-14, May.
    8. Zsolt Kohus & Márton Demeter & László Kun & Eszter Lukács & Katalin Czakó & Gyula Péter Szigeti, 2022. "A Study of the Relation between Byline Positions of Affiliated/Non-Affiliated Authors and the Scientific Impact of European Universities in Times Higher Education World University Rankings," Sustainability, MDPI, vol. 14(20), pages 1-14, October.
    9. Horbach, Serge & Aagaard, Kaare & Schneider, Jesper W., 2021. "Meta-Research: How problematic citing practices distort science," MetaArXiv aqyhg, Center for Open Science.
    10. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    11. A. Velez-Estevez & P. García-Sánchez & J. A. Moral-Munoz & M. J. Cobo, 2022. "Why do papers from international collaborations get more citations? A bibliometric analysis of Library and Information Science papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7517-7555, December.
    12. Ortega, José Luis, 2018. "The life cycle of altmetric impact: A longitudinal study of six metrics from PlumX," Journal of Informetrics, Elsevier, vol. 12(3), pages 579-589.
    13. G. Christopher Crawford & Vitaliy Skorodziyevskiy & Casey J. Frid & Thomas E. Nelson & Zahra Booyavi & Diana M. Hechavarria & Xuanye Li & Paul D. Reynolds & Ehsan Teymourian, 2022. "Advancing Entrepreneurship Theory Through Replication: A Case Study on Contemporary Methodological Challenges, Future Best Practices, and an Entreaty for Communality," Entrepreneurship Theory and Practice, , vol. 46(3), pages 779-799, May.
    14. Liwei Zhang & Jue Wang, 2021. "What affects publications’ popularity on Twitter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9185-9198, November.
    15. Edgar D. Zanotto & Vinicius Carvalho, 2021. "Article age- and field-normalized tools to evaluate scientific impact and momentum," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 2865-2883, April.
    16. Indra Budi & Yaniasih Yaniasih, 2023. "Understanding the meanings of citations using sentiment, role, and citation function classifications," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 735-759, January.
    17. Hyejin Park & Han Woo Park, 2018. "Research evaluation of Asian countries using altmetrics: comparing South Korea, Japan, Taiwan, Singapore, and China," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 771-788, November.
    18. Hasanov, Akram Shavkatovich & Shaiban, Mohammed Sharaf & Al-Freedi, Ajab, 2020. "Forecasting volatility in the petroleum futures markets: A re-examination and extension," Energy Economics, Elsevier, vol. 86(C).
    19. Li, Huixu & Liu, Lanjian & Wang, Xianwen, 2021. "The open access effect in social media exposure of scholarly articles: A matched-pair analysis," Journal of Informetrics, Elsevier, vol. 15(3).
    20. Kahn, Kenneth B. & Candi, Marina, 2021. "Investigating the relationship between innovation strategy and performance," Journal of Business Research, Elsevier, vol. 132(C), pages 56-66.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:2gurz. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.