IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0322696.html
   My bibliography  Save this article

Registered report protocol: Factors associated with inter-rater agreement in grant peer review

Author

Listed:
  • Jan-Ole Hesselberg
  • Pål Ulleberg
  • Øystein Sørensen
  • Knut Inge Fostervold
  • Sigrid Hegna Ingvaldsen
  • Ida Svege

Abstract

Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study aims to examine factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we will explore the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our findings are expected to shed light on whether similarity between reviewers (gender, age), their experience, or expertise correlates with higher agreement. Additionally, we investigate whether characteristics of the applications—such as funding amount, research area, or variability in project size—affect agreement levels. By analyzing applications from diverse disciplines and funding schemes, this study aims to provide a comprehensive understanding of the drivers of inter-rater agreement and their implications for grant peer review reliability. The results will inform improvements to peer review processes, enhancing the fairness and validity of funding decisions. All data and analysis scripts will be publicly available, ensuring transparency and reproducibility.

Suggested Citation

  • Jan-Ole Hesselberg & Pål Ulleberg & Øystein Sørensen & Knut Inge Fostervold & Sigrid Hegna Ingvaldsen & Ida Svege, 2025. "Registered report protocol: Factors associated with inter-rater agreement in grant peer review," PLOS ONE, Public Library of Science, vol. 20(5), pages 1-11, May.
    • Handle: RePEc:plo:pone00:0322696
    DOI: 10.1371/journal.pone.0322696
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0322696
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0322696&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0322696?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gerald Schweiger & Adrian Barnett & Peter van den Besselaar & Lutz Bornmann & Andreas De Block & John P. A. Ioannidis & Ulf Sandström & Stijn Conix, 2024. "The costs of competition in distributing scarce research funds," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 121(50), pages 2407644121-, December.
    2. Marco Seeber, 2020. "How do journals of different rank instruct peer reviewers? Reviewer guidelines in the field of management," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1387-1405, March.
    3. Marco Seeber & Jef Vlegels & Elwin Reimink & Ana Marušić & David G Pina, 2021. "Does reviewing experience reduce disagreement in proposals evaluation? Insights from Marie Skłodowska-Curie and COST Actions [Peer Review of Grant Applications: Criteria Used and Qualitative Study ," Research Evaluation, Oxford University Press, vol. 30(3), pages 349-360.
    4. Gerald Schweiger & Adrian Barnett & Peter van den Besselaar & Lutz Bornmann & Andreas De Block & John P. A. Ioannidis & Ulf Sandstrom & Stijn Conix, 2024. "The Costs of Competition in Distributing Scarce Research Funds," Papers 2403.16934, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2020. "The author–reviewer game," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2409-2431, September.
    2. Bianchi, Federico & García-Costa, Daniel & Grimaldo, Francisco & Squazzoni, Flaminio, 2022. "Measuring the effect of reviewers on manuscript change: A study on a sample of submissions to Royal Society journals (2006–2017)," Journal of Informetrics, Elsevier, vol. 16(3).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0322696. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.