IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2303.16855.html
   My bibliography  Save this paper

Peer Prediction for Peer Review: Designing a Marketplace for Ideas

Author

Listed:
  • Alexander Ugarov

Abstract

The paper describes a potential platform to facilitate academic peer review with emphasis on early-stage research. This platform aims to make peer review more accurate and timely by rewarding reviewers on the basis of peer prediction algorithms. The algorithm uses a variation of Peer Truth Serum for Crowdsourcing (Radanovic et al., 2016) with human raters competing against a machine learning benchmark. We explain how our approach addresses two large productive inefficiencies in science: mismatch between research questions and publication bias. Better peer review for early research creates additional incentives for sharing it, which simplifies matching ideas to teams and makes negative results and p-hacking more visible.

Suggested Citation

  • Alexander Ugarov, 2023. "Peer Prediction for Peer Review: Designing a Marketplace for Ideas," Papers 2303.16855, arXiv.org.
    • Handle: RePEc:arx:papers:2303.16855
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2303.16855
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    2. Robert E. Lucas Jr. & Benjamin Moll, 2014. "Knowledge Growth and the Allocation of Time," Journal of Political Economy, University of Chicago Press, vol. 122(1), pages 1-51.
    3. Marcus Berliant & Masahisa Fujita, 2008. "Knowledge Creation As A Square Dance On The Hilbert Cube," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 49(4), pages 1251-1295, November.
    4. Fiona Murray & Philippe Aghion & Mathias Dewatripont & Julian Kolev & Scott Stern, 2016. "Of Mice and Academics: Examining the Effect of Openness on Innovation," American Economic Journal: Economic Policy, American Economic Association, vol. 8(1), pages 212-252, February.
    5. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    6. Benjamin F. Jones, 2011. "As Science Evolves, How Can Science Policy?," NBER Chapters, in: Innovation Policy and the Economy, Volume 11, pages 103-131, National Bureau of Economic Research, Inc.
    7. Siddarth Srinivasan & Jamie Morgenstern, 2021. "Auctions and Peer Prediction for Academic Peer Review," Papers 2109.00923, arXiv.org, revised May 2023.
    8. Adam Mann, 2016. "The power of prediction markets," Nature, Nature, vol. 538(7625), pages 308-310, October.
    9. Partha, Dasgupta & David, Paul A., 1994. "Toward a new economics of science," Research Policy, Elsevier, vol. 23(5), pages 487-521, September.
    10. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeffrey L. Furman & Florenta Teodoridis, 2020. "Automation, Research Technology, and Researchers’ Trajectories: Evidence from Computer Science and Electrical Engineering," Organization Science, INFORMS, vol. 31(2), pages 330-354, March.
    2. Gans, Joshua S. & Murray, Fiona E. & Stern, Scott, 2017. "Contracting over the disclosure of scientific knowledge: Intellectual property and academic publication," Research Policy, Elsevier, vol. 46(4), pages 820-835.
    3. Mueller-Langer, Frank & Scheufen, Marc & Waelbroeck, Patrick, 2020. "Does online access promote research in developing countries? Empirical evidence from article-level data," Research Policy, Elsevier, vol. 49(2).
    4. Birgitte Andersen & Federica Rossi, 2011. "Intellectual property governance and knowledge creation in UK universities," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 20(8), pages 701-725, September.
    5. Donald R. Davis & Jonathan I. Dingel, 2019. "A Spatial Knowledge Economy," American Economic Review, American Economic Association, vol. 109(1), pages 153-170, January.
    6. Tomoya Mori & Shosei Sakaguchi, 2019. "Creation of knowledge through exchanges of knowledge: Evidence from Japanese patent data," Papers 1908.01256, arXiv.org, revised Aug 2020.
    7. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    8. Simeth, Markus & Raffo, Julio D., 2013. "What makes companies pursue an Open Science strategy?," Research Policy, Elsevier, vol. 42(9), pages 1531-1543.
    9. Fehder, Daniel C. & Murray, Fiona & Stern, Scott, 2014. "Intellectual property rights and the evolution of scientific journals as knowledge platforms," International Journal of Industrial Organization, Elsevier, vol. 36(C), pages 83-94.
    10. Douglas Hanley & Ufuk Akcigit & Nicolas Serrano-Velarde, 2014. "Back to Basics: Basic Research Spillovers, Innovation Policy and Growth," Working Paper 535, Department of Economics, University of Pittsburgh, revised Jan 2014.
    11. Annamaria Conti & Christopher C. Liu, 2014. "The (Changing) Knowledge Production Function: Evidence from the MIT Department of Biology for 1970–2000," NBER Chapters, in: The Changing Frontier: Rethinking Science and Innovation Policy, pages 49-74, National Bureau of Economic Research, Inc.
    12. Dosi, Giovanni & Nelson, Richard R., 2010. "Technical Change and Industrial Dynamics as Evolutionary Processes," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 1, chapter 0, pages 51-127, Elsevier.
    13. Annamaria Conti & Christopher C. Liu, 2014. "The (Changing) Knowledge Production Function: Evidence from the MIT Department of Biology for 1970-2000," NBER Working Papers 20037, National Bureau of Economic Research, Inc.
    14. Larsen, Maria Theresa, 2011. "The implications of academic enterprise for public science: An overview of the empirical evidence," Research Policy, Elsevier, vol. 40(1), pages 6-19, February.
    15. Florenta Teodoridis, 2018. "Understanding Team Knowledge Production: The Interrelated Roles of Technology and Expertise," Management Science, INFORMS, vol. 64(8), pages 3625-3648, August.
    16. Dirk Czarnitzki & Christoph Grimpe & Andrew A. Toole, 2015. "Delay and secrecy: does industry sponsorship jeopardize disclosure of academic research?," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 24(1), pages 251-279.
    17. Arianna Martinelli & Elena Romito, 2019. "When authors become inventors: an empirical analysis on patent-paper pairs in medical research," LEM Papers Series 2019/32, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    18. Boudreau, Kevin J. & Lakhani, Karim R., 2015. "“Open” disclosure of innovations, incentives and follow-on reuse: Theory on processes of cumulative innovation and a field experiment in computational biology," Research Policy, Elsevier, vol. 44(1), pages 4-19.
    19. Balland, Pierre-Alexandre & Boschma, Ron, 2022. "Do scientific capabilities in specific domains matter for technological diversification in European regions?," Research Policy, Elsevier, vol. 51(10).
    20. Julia M. Puaschunder, 2019. "Artificial Intelligence Market Disruption," Proceedings of the 13th International RAIS Conference, June 10-11, 2019 01 JP, Research Association for Interdisciplinary Studies.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2303.16855. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.