IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2011.07797.html
   My bibliography  Save this paper

Double blind vs. open review: an evolutionary game logit-simulating the behavior of authors and reviewers

Author

Listed:
  • Mantas Radzvilas
  • Francesco De Pretis
  • William Peden
  • Daniele Tortoli
  • Barbara Osimani

Abstract

Despite the tremendous successes of science in providing knowledge and technologies, the Replication Crisis has highlighted that scientific institutions have much room for improvement. Peer-review is one target of criticism and suggested reforms. However, despite numerous controversies peer review systems, plus the obvious complexity of the incentives affecting the decisions of authors and reviewers, there is very little systematic and strategic analysis of peer-review systems. In this paper, we begin to address this feature of the peer-review literature by applying the tools of game theory. We use simulations to develop an evolutionary model based around a game played by authors and reviewers, before exploring some of its tendencies. In particular, we examine the relative impact of double-blind peer-review and open review on incentivising reviewer effort under a variety of parameters. We also compare (a) the impact of one review system versus another with (b) other alterations, such as higher costs of reviewing. We find that is no reliable difference between peer-review systems in our model. Furthermore, under some conditions, higher payoffs for good reviewing can lead to less (rather than more) author effort under open review. Finally, compared to the other parameters that we vary, it is the exogenous utility of author effort that makes an important and reliable difference in our model, which raises the possibility that peer-review might not be an important target for institutional reforms.

Suggested Citation

  • Mantas Radzvilas & Francesco De Pretis & William Peden & Daniele Tortoli & Barbara Osimani, 2020. "Double blind vs. open review: an evolutionary game logit-simulating the behavior of authors and reviewers," Papers 2011.07797, arXiv.org.
    • Handle: RePEc:arx:papers:2011.07797
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2011.07797
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Francesc Carreras & Josep Freixas & Antonio Magaña, 2016. "Dimension, egalitarianism and decisiveness of European voting systems," Operations Research and Decisions, Wroclaw University of Science and Technology, Faculty of Management, vol. 26(2), pages 31-52.
    3. Hofbauer, Josef & Sandholm, William H., 2007. "Evolution in games with randomly disturbed payoffs," Journal of Economic Theory, Elsevier, vol. 132(1), pages 47-69, January.
    4. Daniel S. Hamermesh, 2013. "Six Decades of Top Economics Publishing: Who and How?," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 162-172, March.
    5. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    6. Marianne A. Ferber & Michelle Teiman, 1980. "Are Women Economists at a Disadvantage in Publishing Journal Articles?," Eastern Economic Journal, Eastern Economic Association, vol. 6(3-4), pages 189-193, Aug-Oct.
    7. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    8. Curtis R. Taylor & Huseyin Yildirim, 2011. "Subjective Performance and the Value of Blind Evaluation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 78(2), pages 762-794.
    9. Izquierdo, Luis R. & Izquierdo, Segismundo S. & Sandholm, William H., 2019. "An introduction to ABED: Agent-based simulation of evolutionary game dynamics," Games and Economic Behavior, Elsevier, vol. 118(C), pages 434-462.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mantas Radzvilas & Francesco De Pretis & William Peden & Daniele Tortoli & Barbara Osimani, 2023. "Incentives for Research Effort: An Evolutionary Model of Publication Markets with Double-Blind and Open Review," Computational Economics, Springer;Society for Computational Economics, vol. 61(4), pages 1433-1476, April.
    2. Valérie Orozco & Christophe Bontemps & Élise Maigné & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.
    3. Sascha Baghestanian & Sergey V. Popov, 2018. "On publication, refereeing and working hard," Canadian Journal of Economics, Canadian Economics Association, vol. 51(4), pages 1419-1459, November.
    4. Valérie Orozco & Christophe Bontemps & Elise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐Marc Rousselle, 2020. "How To Make A Pie: Reproducible Research For Empirical Economics And Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(5), pages 1134-1169, December.
    5. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    6. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    7. Uwe Hassler & Marc‐Oliver Pohle, 2022. "Unlucky Number 13? Manipulating Evidence Subject to Snooping," International Statistical Review, International Statistical Institute, vol. 90(2), pages 397-410, August.
    8. Allen Bellas & Lea-Rachel Kosnik, 2019. "Which leading journal leads? Idea diffusion in economics research journals," Empirical Economics, Springer, vol. 57(3), pages 901-921, September.
    9. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    10. Önder Ali Sina & Yilmazkuday Hakan, 2020. "Thirty-Five Years of Peer-Reviewed Publishing by North American Economics PhDs: Quantity, Quality, and Beyond," Open Economics, De Gruyter, vol. 3(1), pages 70-85, January.
    11. Syed Hasan & Robert Breunig, 2021. "Article length and citation outcomes," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7583-7608, September.
    12. Fabian Scheidegger & Andre Briviba & Bruno S. Frey, 2023. "Behind the curtains of academic publishing: strategic responses of economists and business scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4765-4790, August.
    13. Claude Diebolt & Michael Haupert, 2021. "The Role of Cliometrics in History and Economics," Working Papers of BETA 2021-26, Bureau d'Economie Théorique et Appliquée, UDS, Strasbourg.
    14. Paldam Martin, 2021. "Methods Used in Economic Research: An Empirical Study of Trends and Levels," Economics - The Open-Access, Open-Assessment Journal, De Gruyter, vol. 15(1), pages 28-42, January.
    15. Lea†Rachel Kosnik, 2018. "A Survey Of Jel Codes: What Do They Mean And Are They Used Consistently?," Journal of Economic Surveys, Wiley Blackwell, vol. 32(1), pages 249-272, February.
    16. Thiago Dumont Oliveira & Marwil J. Dávila-Fernández, 2020. "From modelmania to datanomics? The rise of mathematical and quantitative methods in three top economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 51-70, April.
    17. Nicolas Vallois & Dorian Jullien, 2017. "Estimating Rationality in Economics: A History of Statistical Methods in Experimental Economics," Working Papers halshs-01651070, HAL.
    18. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    19. Grażyna Bukowska & Jan Fałkowski & Beata Łopaciuk-Gonczaryk, 2014. "Teaming up or writing alone - authorship strategies in leading Polish economic journals," Working Papers 2014-29, Faculty of Economic Sciences, University of Warsaw.
    20. David Spiegelhalter, 2017. "Trust in numbers," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(4), pages 948-965, October.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2011.07797. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.