IDEAS home Printed from https://ideas.repec.org/a/plo/pbio00/3002238.html
   My bibliography  Save this article

Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews

Author

Listed:
  • Anna Severin
  • Michaela Strinzel
  • Matthias Egger
  • Tiago Barros
  • Alexander Sokolov
  • Julia Vilstrup Mouatt
  • Stefan Müller

Abstract

The Journal Impact Factor is often used as a proxy measure for journal quality, but the empirical evidence is scarce. In particular, it is unclear how peer review characteristics for a journal relate to its impact factor. We analysed 10,000 peer review reports submitted to 1,644 biomedical journals with impact factors ranging from 0.21 to 74.7. Two researchers hand-coded sentences using categories of content related to the thoroughness of the review (Materials and Methods, Presentation and Reporting, Results and Discussion, Importance and Relevance) and helpfulness (Suggestion and Solution, Examples, Praise, Criticism). We fine-tuned and validated transformer machine learning language models to classify sentences. We then examined the association between the number and percentage of sentences addressing different content categories and 10 groups defined by the Journal Impact Factor. The median length of reviews increased with higher impact factor, from 185 words (group 1) to 387 words (group 10). The percentage of sentences addressing Materials and Methods was greater in the highest Journal Impact Factor journals than in the lowest Journal Impact Factor group. The results for Presentation and Reporting went in the opposite direction, with the highest Journal Impact Factor journals giving less emphasis to such content. For helpfulness, reviews for higher impact factor journals devoted relatively less attention to Suggestion and Solution than lower impact factor journals. In conclusion, peer review in journals with higher impact factors tends to be more thorough, particularly in addressing study methods while giving relatively less emphasis to presentation or suggesting solutions. Differences were modest and variability high, indicating that the Journal Impact Factor is a bad predictor of the quality of peer review of an individual manuscript.An analysis of the content of 10,000 peer review reports reveals that reports submitted to journals with higher impact factors pay more attention to the materials and methods of a study but less attention to presentation and reporting, whereas journals with low impact factors provide more suggestions, solutions and examples.

Suggested Citation

  • Anna Severin & Michaela Strinzel & Matthias Egger & Tiago Barros & Alexander Sokolov & Julia Vilstrup Mouatt & Stefan Müller, 2023. "Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews," PLOS Biology, Public Library of Science, vol. 21(8), pages 1-18, August.
    • Handle: RePEc:plo:pbio00:3002238
    DOI: 10.1371/journal.pbio.3002238
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002238
    Download Restriction: no
    File URL: https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.3002238&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pbio.3002238?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Agnes Grudniewicz & David Moher & Kelly D. Cobey & Gregory L. Bryson & Samantha Cukier & Kristiann Allen & Clare Ardern & Lesley Balcom & Tiago Barros & Monica Berger & Jairo Buitrago Ciro & Lucia Cug, 2019. "Predatory journals: no definition, no defence," Nature, Nature, vol. 576(7786), pages 210-212, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Edré Moreira & Wagner Meira & Marcos André Gonçalves & Alberto H. F. Laender, 2023. "The rise of hyperprolific authors in computer science: characterization and implications," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 2945-2974, May.
    2. Jaime A. Teixeira da Silva & Quan-Hoang Vuong, 2021. "The right to refuse unwanted citations: rethinking the culture of science around the citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5355-5360, June.
    3. Onur Öztürk & Zehra Taşkın, 2024. "How metric-based performance evaluation systems fuel the growth of questionable publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(5), pages 2729-2748, May.
    4. Kumar, Satish & Sahoo, Saumyaranjan & Lim, Weng Marc & Kraus, Sascha & Bamel, Umesh, 2022. "Fuzzy-set qualitative comparative analysis (fsQCA) in business and management research: A contemporary overview," Technological Forecasting and Social Change, Elsevier, vol. 178(C).
    5. Carlos Ruiz-Frutos & Mónica Ortega-Moreno & Adriano Dias & João Marcos Bernardes & Juan Jesús García-Iglesias & Juan Gómez-Salgado, 2020. "Information on COVID-19 and Psychological Distress in a Sample of Non-Health Workers during the Pandemic Period," IJERPH, MDPI, vol. 17(19), pages 1-18, September.
    6. repec:hal:ciredw:hal-04093198 is not listed on IDEAS
    7. Evgeny N. Smirnov & Sergey A. Lukyanov, 2022. "Junk journals: Scientometrics vs Science," Upravlenets, Ural State University of Economics, vol. 13(4), pages 83-95, September.
    8. repec:osf:osfxxx:xr8mv_v1 is not listed on IDEAS
    9. Jaime A. Teixeira da Silva & Daniel J. Dunleavy & Mina Moradzadeh & Joshua Eykens, 2021. "A credit-like rating system to determine the legitimacy of scientific journals and publishers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8589-8616, October.
    10. Libor Ansorge, 2023. "The right to reject an unwanted citations: do we need it?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 4147-4150, July.
    11. repec:hal:wpaper:hal-04093198 is not listed on IDEAS
    12. Minh Ha-Duong, 2023. "Against predatory publishing: the IAP report results," Post-Print hal-04093198, HAL.
    13. Philip Darbyshire & Mark Hayter & Kate Frazer & Robin Ion & Debra Jackson, 2020. "Hitting rock bottom: The descent from predatory journals and conferences to the predatory PhD," Journal of Clinical Nursing, John Wiley & Sons, vol. 29(23-24), pages 4425-4428, December.
    14. Uztürk, Deniz & Büyüközkan, Gülçin, 2024. "Industry 4.0 technologies in Smart Agriculture: A review and a Technology Assessment Model proposition," Technological Forecasting and Social Change, Elsevier, vol. 208(C).
    15. Emanuel Kulczycki & Marek Hołowiecki & Zehra Taşkın & Franciszek Krawczyk, 2021. "Citation patterns between impact-factor and questionable journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8541-8560, October.
    16. Heike Schuler & Valeria Bonapersona & Marian Joëls & R Angela Sarabdjitsingh, 2022. "Effects of early life adversity on immediate early gene expression: Systematic review and 3-level meta-analysis of rodent studies," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-21, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pbio00:3002238. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosbiology (email available below). General contact details of provider: https://journals.plos.org/plosbiology/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.