IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i3s1751157722000682.html
   My bibliography  Save this article

Measuring the effect of reviewers on manuscript change: A study on a sample of submissions to Royal Society journals (2006–2017)

Author

Listed:
  • Bianchi, Federico
  • García-Costa, Daniel
  • Grimaldo, Francisco
  • Squazzoni, Flaminio

Abstract

Peer review is key for public trust of academic journals. It ensures that only rigorous research is published but also helps authors to increase the value of their manuscripts through feedback from reviewers. However, measuring the developmental value of peer review is difficult as it requires fine-grained manuscript data on various stages of the editorial process, which are rarely available. To fill this gap, we accessed complete data from Royal Society journals from 2006 to 2017, and measured manuscript changes during peer review from their initial submissions. We then estimated the effect of the number of reviewers and the evaluation of reviewers on manuscript development and their citations after publication. We found that the number of reviewers had an almost linear effect on manuscript change although with decreasing marginal effects whenever more than two reviewers were involved. This effect did not depend on the initial quality of manuscripts. We also found that changes due to reviewers tended to increase a manuscript’s probability of being cited at least once after publication. While our findings show the multiple functions of peer review for manuscript development, research with larger and more representative journal samples is needed to develop more systematic measures that reflect the complexity of peer review.

Suggested Citation

  • Bianchi, Federico & García-Costa, Daniel & Grimaldo, Francisco & Squazzoni, Flaminio, 2022. "Measuring the effect of reviewers on manuscript change: A study on a sample of submissions to Royal Society journals (2006–2017)," Journal of Informetrics, Elsevier, vol. 16(3).
    • Handle: RePEc:eee:infome:v:16:y:2022:i:3:s1751157722000682
    DOI: 10.1016/j.joi.2022.101316
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000682
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2022.101316?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Harms, P. D. & Credé, Marcus, 2020. "Bringing the review process into the 21st century: Post-publication peer review," Industrial and Organizational Psychology, Cambridge University Press, vol. 13(1), pages 51-53, March.
    2. José Luis Ortega, 2017. "Are peer-review activities related to reviewer bibliometric performance? A scientometric analysis of Publons," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 947-962, August.
    3. Marco Seeber, 2020. "How do journals of different rank instruct peer reviewers? Reviewer guidelines in the field of management," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1387-1405, March.
    4. Hilbe,Joseph M., 2014. "Modeling Count Data," Cambridge Books, Cambridge University Press, number 9781107028333.
    5. Bravo, Giangiacomo & Farjam, Mike & Grimaldo Moreno, Francisco & Birukou, Aliaksandr & Squazzoni, Flaminio, 2018. "Hidden connections: Network effects on editorial decisions in four computer science journals," Journal of Informetrics, Elsevier, vol. 12(1), pages 101-112.
    6. Flaminio Squazzoni & Petra Ahrweiler & Tiago Barros & Federico Bianchi & Aliaksandr Birukou & Harry J. J. Blom & Giangiacomo Bravo & Stephen Cowley & Virginia Dignum & Pierpaolo Dondio & Francisco Gri, 2020. "Unlock ways to share data on peer review," Nature, Nature, vol. 578(7796), pages 512-514, February.
    7. Janine Huisman & Jeroen Smits, 2017. "Duration and quality of the peer review process: the author’s perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 633-650, October.
    8. J. Rigby & D. Cox & K. Julian, 2018. "Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1087-1105, March.
    9. Flaminio Squazzoni & Francisco Grimaldo & Ana Marušić, 2017. "Publishing: Journals could share peer-review data," Nature, Nature, vol. 546(7658), pages 352-352, June.
    10. Liu, Meijun & Hu, Xiao & Wang, Yuandi & Shi, Dongbo, 2018. "Survive or perish: Investigating the life cycle of academic journals from 1950 to 2013 using survival analysis methods," Journal of Informetrics, Elsevier, vol. 12(1), pages 344-364.
    11. Tammy Harris & Joseph M. Hilbe & James W. Hardin, 2014. "Modeling count data with generalized distributions," Stata Journal, StataCorp LP, vol. 14(3), pages 562-579, September.
    12. Lugosi, Peter, 2021. "The value creation cycle of peer review," Annals of Tourism Research, Elsevier, vol. 86(C).
    13. Niccolò Casnici & Francisco Grimaldo & Nigel Gilbert & Flaminio Squazzoni, 2017. "Attitudes of referees in a multidisciplinary journal: An empirical analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(7), pages 1763-1771, July.
    14. Adam Eyre-Walker & Nina Stoletzki, 2013. "The Assessment of Science: The Relative Merits of Post-Publication Review, the Impact Factor, and the Number of Citations," PLOS Biology, Public Library of Science, vol. 11(10), pages 1-8, October.
    15. Dondio, Pierpaolo & Casnici, Niccolò & Grimaldo, Francisco & Gilbert, Nigel & Squazzoni, Flaminio, 2019. "The “invisible hand” of peer review: The implications of author-referee networks on peer review in a scholarly journal," Journal of Informetrics, Elsevier, vol. 13(2), pages 708-716.
    16. Taşkın, Zehra & Doğan, Güleda & Kulczycki, Emanuel & Zuccala, Alesia Ann, 2021. "Self-Citation Patterns of Journals Indexed in the Journal Citation Reports," Journal of Informetrics, Elsevier, vol. 15(4).
    17. Federico Bianchi & Francisco Grimaldo & Giangiacomo Bravo & Flaminio Squazzoni, 2018. "The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1401-1420, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    2. Akbaritabar, Aliakbar & Stephen, Dimity & Squazzoni, Flaminio, 2022. "A study of referencing changes in preprint-publication pairs across multiple fields," Journal of Informetrics, Elsevier, vol. 16(2).
    3. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    4. Wu, Jiang & Ou, Guiyan & Liu, Xiaohui & Dong, Ke, 2022. "How does academic education background affect top researchers’ performance? Evidence from the field of artificial intelligence," Journal of Informetrics, Elsevier, vol. 16(2).
    5. Mutz, Rüdiger & Daniel, Hans-Dieter, 2018. "The bibliometric quotient (BQ), or how to measure a researcher’s performance capacity: A Bayesian Poisson Rasch model," Journal of Informetrics, Elsevier, vol. 12(4), pages 1282-1295.
    6. Chiara Bocci & Laura Grassini & Emilia Rocco, 2021. "A multiple inflated negative binomial hurdle regression model: analysis of the Italians’ tourism behaviour during the Great Recession," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(4), pages 1109-1133, October.
    7. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.
    8. Maniezzo, Vittorio & Boschetti, Marco A. & Gutjahr, Walter J., 2021. "Stochastic premarshalling of block stacking warehouses," Omega, Elsevier, vol. 102(C).
    9. Geng, Xi & Xia, Aihua, 2022. "When is the Conway–Maxwell–Poisson distribution infinitely divisible?," Statistics & Probability Letters, Elsevier, vol. 181(C).
    10. Smith, David M. & Faddy, Malcolm J., 2016. "Mean and Variance Modeling of Under- and Overdispersed Count Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 69(i06).
    11. de Rezende, Rafael & Egert, Katharina & Marin, Ignacio & Thompson, Guilherme, 2022. "A white-boxed ISSM approach to estimate uncertainty distributions of Walmart sales," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1460-1467.
    12. Andries, Petra & Hünermund, Paul, 2020. "Firm-level effects of staged investments in innovation: The moderating role of resource availability," Research Policy, Elsevier, vol. 49(7).
    13. Lin, Jun-You, 2017. "Balancing industry collaboration and academic innovation: The contingent role of collaboration-specific attributes," Technological Forecasting and Social Change, Elsevier, vol. 123(C), pages 216-228.
    14. Di Novi, Cinzia & Leporatti, Lucia & Levaggi, Rosella & Montefiori, Marcello, 2022. "Adherence during COVID-19: The role of aging and socio-economics status in shaping drug utilization," Journal of Economic Behavior & Organization, Elsevier, vol. 204(C), pages 1-14.
    15. Hu, Feng & Bijmolt, Tammo H.A. & Huizingh, Eelko K.R.E., 2020. "The impact of innovation contest briefs on the quality of solvers and solutions," Technovation, Elsevier, vol. 90.
    16. Sinclair, Michael & Ghermandi, Andrea & Signorello, Giovanni & Giuffrida, Laura & De Salvo, Maria, 2022. "Valuing Recreation in Italy's Protected Areas Using Spatial Big Data," Ecological Economics, Elsevier, vol. 200(C).
    17. Hilbe, Joseph M., 2015. "The New Statistics with R: An Introduction for Biologists," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 68(b01).
    18. Daisuke Sakai, 2019. "Who is peer reviewed? Comparing publication patterns of peer-reviewed and non-peer-reviewed papers in Japanese political science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 65-80, October.
    19. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2020. "The author–reviewer game," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2409-2431, September.
    20. Corona Francisco & Wiper Michael Peter & Horrillo Juan de Dios Tena, 2017. "On the importance of the probabilistic model in identifying the most decisive games in a tournament," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 13(1), pages 11-23, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:3:s1751157722000682. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.