IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v18y2024i3s1751157724000683.html
   My bibliography  Save this article

Development and application of a comprehensive glossary for the identification of statistical and methodological concepts in peer review reports

Author

Listed:
  • Buljan, Ivan
  • Garcia-Costa, Daniel
  • Grimaldo, Francisco
  • Klein, Richard A.
  • Bakker, Marjan
  • Marušić, Ana

Abstract

The assessment of problems identified by peer researchers during peer review is difficult because the content of these reports is typically confidential. The current study sought to construct and apply a glossary for the identification of methodological and statistical concepts mentioned in peer review reports. Three assessors created a list of 1,036 different terms in 19 categories. The glossary was tested on the confidential PEERE database, a sample of 496,928 peer review reports from various scientific disciplines. The most frequently mentioned terms were related to data presentation (found in 40.3 % of the reports) and parametric descriptive statistics (33.3 %). Review reports suggesting a rejection were more likely to mention methodological issues, whereas statistical issues were raised more frequently in review reports recommending revisions. Across disciplines, methodological issues were more frequently mentioned in social sciences (64.1 %), while health and medical sciences were more predictive for the identification of statistical issues (40.1 %). Female reviewers identified more statistical issues compared to male reviewers. These results indicate that the glossary could be used as an additional tool for the assessment of the content of peer review reports and for understanding what help authors may need in writing research articles.

Suggested Citation

  • Buljan, Ivan & Garcia-Costa, Daniel & Grimaldo, Francisco & Klein, Richard A. & Bakker, Marjan & Marušić, Ana, 2024. "Development and application of a comprehensive glossary for the identification of statistical and methodological concepts in peer review reports," Journal of Informetrics, Elsevier, vol. 18(3).
    • Handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000683
    DOI: 10.1016/j.joi.2024.101555
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157724000683
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2024.101555?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sun, Zhuanlan & Clark Cao, C. & Ma, Chao & Li, Yiwei, 2023. "The academic status of reviewers predicts their language use," Journal of Informetrics, Elsevier, vol. 17(4).
    2. Flaminio Squazzoni & Petra Ahrweiler & Tiago Barros & Federico Bianchi & Aliaksandr Birukou & Harry J. J. Blom & Giangiacomo Bravo & Stephen Cowley & Virginia Dignum & Pierpaolo Dondio & Francisco Gri, 2020. "Unlock ways to share data on peer review," Nature, Nature, vol. 578(7796), pages 512-514, February.
    3. Taylor Bolt & Jason S Nomi & Danilo Bzdok & Lucina Q Uddin, 2021. "Educating the future generation of researchers: A cross-disciplinary survey of trends in analysis methods," PLOS Biology, Public Library of Science, vol. 19(7), pages 1-23, July.
    4. Tirthankar Ghosal & Sandeep Kumar & Prabhat Kumar Bharti & Asif Ekbal, 2022. "Peer review analyze: A novel benchmark resource for computational analysis of peer reviews," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-29, January.
    5. Alessandro Checco & Lorenzo Bracciale & Pierpaolo Loreti & Stephen Pinfield & Giuseppe Bianchi, 2021. "AI-assisted peer review," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-11, December.
    6. Serge P J M Horbach & Freek J W Oude Maatman & Willem Halffman & Wytske M Hepkema, 2022. "Automated citation recommendation tools encourage questionable citations," Research Evaluation, Oxford University Press, vol. 31(3), pages 321-325.
    7. Weiss, Matthias & Nair, Lakshmi B. & Hoorani, Bareerah H. & Gibbert, Michael & Hoegl, Martin, 2023. "Transparency of reporting practices in quantitative field studies: The transparency sweet spot for article citations," Journal of Informetrics, Elsevier, vol. 17(2).
    8. Sizo, Amanda & Lino, Adriano & Reis, Luis Paulo & Rocha, Álvaro, 2019. "An overview of assessing the quality of peer review reports of scientific articles," International Journal of Information Management, Elsevier, vol. 46(C), pages 286-293.
    9. Giangiacomo Bravo & Francisco Grimaldo & Emilia López-Iñesta & Bahar Mehmani & Flaminio Squazzoni, 2019. "The effect of publishing peer review reports on referee behavior in five scholarly journals," Nature Communications, Nature, vol. 10(1), pages 1-8, December.
    10. Tom E Hardwicke & Steven N Goodman, 2020. "How often do leading biomedical journals use statistical experts to evaluate statistical methods? The results of a survey," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-12, October.
    11. Sam Parsons & Flávio Azevedo & Mahmoud M. Elsherif & Samuel Guay & Owen N. Shahim & Gisela H. Govaart & Emma Norris & Aoife O’Mahony & Adam J. Parker & Ana Todorovic & Charlotte R. Pennington & Elias , 2022. "A community-sourced glossary of open scholarship terms," Nature Human Behaviour, Nature, vol. 6(3), pages 312-318, March.
    12. Jurgen Huber & Sabiou M. Inoua & Rudolf Kerschbamer & Christian Konig-Kersting & Stefan Palan & Vernon L. Smith, 2022. "Nobel and Novice: Author Prominence Affects Peer Review," Working Papers 22-15, Chapman University, Economic Science Institute.
    13. Andrijana Perković Paloš & Antonija Mijatović & Ivan Buljan & Daniel Garcia-Costa & Elena Álvarez-García & Francisco Grimaldo & Ana Marušić, 2023. "Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4707-4729, August.
    14. Charles W. Fox, 2017. "Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 465-477, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sun, Zhuanlan, 2024. "Textual features of peer review predict top-cited papers: An interpretable machine learning perspective," Journal of Informetrics, Elsevier, vol. 18(2).
    2. Sun, Zhuanlan & Pang, Ka Lok & Li, Yiwei, 2024. "The fading of status bias during the open peer review process," Journal of Informetrics, Elsevier, vol. 18(3).
    3. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    4. Sun, Zhuanlan & Clark Cao, C. & Ma, Chao & Li, Yiwei, 2023. "The academic status of reviewers predicts their language use," Journal of Informetrics, Elsevier, vol. 17(4).
    5. Wenqing Wu & Haixu Xi & Chengzhi Zhang, 2024. "Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4109-4135, July.
    6. Sun, Zhuanlan & He, Dongjin & Li, Yiwei, 2024. "How the readability of manuscript before journal submission advantages peer review process: Evidence from biomedical scientific publications," Journal of Informetrics, Elsevier, vol. 18(3).
    7. Ivana Drvenica & Giangiacomo Bravo & Lucija Vejmelka & Aleksandar Dekanski & Olgica Nedić, 2018. "Peer Review of Reviewers: The Author’s Perspective," Publications, MDPI, vol. 7(1), pages 1-10, December.
    8. Houqiang Yu & Yian Liang & Yinghua Xie, 2024. "Can peer review accolade awards motivate reviewers? A large-scale quasi-natural experiment," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-13, December.
    9. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.
    10. Chunli Wei & Jingyi Zhao & Jue Ni & Jiang Li, 2023. "What does open peer review bring to scientific articles? Evidence from PLoS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 2763-2776, May.
    11. Feliciani, Thomas & Morreau, Michael & Luo, Junwen & Lucas, Pablo & Shankar, Kalpana, 2022. "Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model," Research Policy, Elsevier, vol. 51(4).
    12. Roggenkamp, Hauke C., 2024. "Revisiting ‘Growth and Inequality in Public Good Provision’—Reproducing and Generalizing Through Inconvenient Online Experimentation," OSF Preprints 6rn97, Center for Open Science.
    13. Tol, Richard S.J., 2023. "Nobel begets Nobel in economics," Journal of Informetrics, Elsevier, vol. 17(4).
    14. Hou, Li & Wu, Qiang & Xie, Yundong, 2024. "Does open identity of peer reviewers positively relate to citations?," Journal of Informetrics, Elsevier, vol. 18(1).
    15. Howell, Bronwyn E. & Potgieter, Petrus H., 2023. "AI-generated lemons: a sour outlook for content producers?," 32nd European Regional ITS Conference, Madrid 2023: Realising the digital decade in the European Union – Easier said than done? 277971, International Telecommunications Society (ITS).
    16. Zhuanlan Sun & C. Clark Cao & Sheng Liu & Yiwei Li & Chao Ma, 2024. "Behavioral consequences of second-person pronouns in written communications between authors and reviewers of scientific papers," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    17. Ying He & Kun Tian & Xiaoran Xu, 2023. "A validation study on the factors affecting the practice modes of open peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 587-607, January.
    18. Bosworth, Steven J. & Della Giusta, Marina, 2024. "When Matthew Met Larry: Explaining the Persistence of Gender Underrepresentation in High Status Organizations," IZA Discussion Papers 17460, Institute of Labor Economics (IZA).
    19. Hajkowicz, Stefan & Naughtin, Claire & Sanderson, Conrad & Schleiger, Emma & Karimi, Sarvnaz & Bratanova, Alexandra & Bednarz, Tomasz, 2022. "Artificial intelligence for science – adoption trends and future development pathways," MPRA Paper 115464, University Library of Munich, Germany.
    20. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000683. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.