IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v17y2023i4s1751157723000743.html
   My bibliography  Save this article

The academic status of reviewers predicts their language use

Author

Listed:
  • Sun, Zhuanlan
  • Clark Cao, C.
  • Ma, Chao
  • Li, Yiwei

Abstract

Peer review plays an essential role in scientific research, but the influence of reviewers' academic status is often overlooked during this process. By accessing peer review reports, in this study we empirically investigate this effect. Specifically, we analyzed 2,580 peer review histories from eLife submissions between 2016 and 2021 to examine the relationship between reviewers’ academic status and their language usage in the first round of peer review. We focused on two types of language features: emotional features (e.g., positivity and subjectivity) and linguistic features (e.g., number of long words and complex words). Our findings revealed no significant reviewer bias of academic status, such that the reviewers’ comments with emotional features were not significantly associated with reviewers’ awareness of being more prestigious than the last corresponding author of the manuscripts. More accomplished reviewers, however, were more likely to use longer and more complex words. Additionally, the results of linguistic features remained robust in the group where the last author served as the last corresponding author. Overall, our findings suggest that the quality of peer review remains the primary consideration for reviewers when evaluating submissions. These results have significant implications for open peer review practices and the fair assessment of the peer review process.

Suggested Citation

  • Sun, Zhuanlan & Clark Cao, C. & Ma, Chao & Li, Yiwei, 2023. "The academic status of reviewers predicts their language use," Journal of Informetrics, Elsevier, vol. 17(4).
    • Handle: RePEc:eee:infome:v:17:y:2023:i:4:s1751157723000743
    DOI: 10.1016/j.joi.2023.101449
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157723000743
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2023.101449?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Oecd, 2008. "DAC Peer Review of Greece," OECD Journal on Development, OECD Publishing, vol. 7(4), pages 7-93.
    2. Shan Jiang, 2021. "Understanding authors' psychological reactions to peer reviews: a text mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6085-6103, July.
    3. Oecd, 2008. "DAC Peer Review of the United Kingdom," OECD Journal on Development, OECD Publishing, vol. 7(3), pages 7-114.
    4. Oecd, 2008. "DAC Peer Review of the Netherlands," OECD Journal on Development, OECD Publishing, vol. 7(3), pages 115-214.
    5. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    6. Yun Wei Zhao & Chi-Hung Chi & Willem-Jan Heuvel, 2015. "Imperfect referees: Reducing the impact of multiple biases in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2340-2356, November.
    7. Giangiacomo Bravo & Francisco Grimaldo & Emilia López-Iñesta & Bahar Mehmani & Flaminio Squazzoni, 2019. "The effect of publishing peer review reports on referee behavior in five scholarly journals," Nature Communications, Nature, vol. 10(1), pages 1-8, December.
    8. Oecd, 2008. "DAC Peer Review of Denmark," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 7-125.
    9. Jürgen Huber & Sabiou Inoua & Rudolf Kerschbamer & Christian König-Kersting & Stefan Palan & Vernon L. Smith, 2022. "Nobel and novice: Author prominence affects peer review," Working Paper Series, Social and Economic Sciences 2022-01, Faculty of Social and Economic Sciences, Karl-Franzens-University Graz.
    10. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    11. Oecd, 2008. "DAC Peer Review of the European Community," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 127-261.
    12. Jue Ni & Zhenyue Zhao & Yupo Shao & Shuo Liu & Wanlin Li & Yaoze Zhuang & Junmo Qu & Yu Cao & Nayuan Lian & Jiang Li, 2021. "The influence of opening up peer review on the citations of journal articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9393-9404, December.
    13. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    14. Costas, Rodrigo & Bordons, María, 2007. "The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level," Journal of Informetrics, Elsevier, vol. 1(3), pages 193-203.
    15. Pauline Mattsson & Carl Johan Sundberg & Patrice Laget, 2011. "Is correspondence reflected in the author position? A bibliometric study of the relation between corresponding author and byline position," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(1), pages 99-105, April.
    16. Wei Huang, 2015. "DO ABCs GET MORE CITATIONS THAN XYZs?," Economic Inquiry, Western Economic Association International, vol. 53(1), pages 773-789, January.
    17. Zhou-min Yuan & Mingxin Yao, 2022. "Is academic writing becoming more positive? A large-scale diachronic case study of Science research articles across 25 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6191-6207, November.
    18. Sun, Zhuanlan & Liu, Sheng & Li, Yiwei & Ma, Chao, 2023. "Expedited editorial decision in COVID-19 pandemic," Journal of Informetrics, Elsevier, vol. 17(1).
    19. Isabelle Régner & Catherine Thinus-Blanc & Agnès Netter & Toni Schmader & Pascal Huguet, 2019. "Committees with implicit biases promote fewer women when they do not believe gender bias exists," Nature Human Behaviour, Nature, vol. 3(11), pages 1171-1179, November.
    20. Lutz Bornmann & Hans-Dieter Daniel, 2005. "Does the h-index for ranking of scientists really work?," Scientometrics, Springer;Akadémiai Kiadó, vol. 65(3), pages 391-392, December.
    21. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    22. Oecd, 2008. "DAC Peer Review of Canada," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 263-387.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    2. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Aleksandar Dekanski, 2020. "The hurdles of academic publishing from the perspective of journal editors: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 115-133, October.
    3. Zhao, Zhi-Dan & Chen, Jiahao & Lu, Yichuan & Zhao, Na & Jiang, Dazhi & Wang, Bing-Hong, 2021. "Dynamic patterns of open review process," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 582(C).
    4. Besim Bilalli & Rana Faisal Munir & Alberto Abelló, 2021. "A framework for assessing the peer review duration of journals: case study in computer science," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 545-563, January.
    5. Siddarth Srinivasan & Jamie Morgenstern, 2021. "Auctions and Peer Prediction for Academic Peer Review," Papers 2109.00923, arXiv.org, revised May 2023.
    6. Stephen A Gallo & Afton S Carpenter & Scott R Glisson, 2013. "Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-9, August.
    7. Hou, Li & Wu, Qiang & Xie, Yundong, 2024. "Does open identity of peer reviewers positively relate to citations?," Journal of Informetrics, Elsevier, vol. 18(1).
    8. José Cendejas Bueno & Cecilia Font de Villanueva, 2015. "Convergence of inflation with a common cycle: estimating and modelling Spanish historical inflation from the 16th to the 18th centuries," Empirical Economics, Springer, vol. 48(4), pages 1643-1665, June.
    9. Lucas Rodriguez Forti & Luiz A. Solino & Judit K. Szabo, 2021. "Trade-off between urgency and reduced editorial capacity affect publication speed in ecological and medical journals during 2020," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-9, December.
    10. Jürgen Janger & Nicole Schmidt & Anna Strauss, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664.
    11. Louise Bedsworth, 2012. "California’s local health agencies and the state’s climate adaptation strategy," Climatic Change, Springer, vol. 111(1), pages 119-133, March.
    12. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    13. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.
    14. Zhentao Liang & Jin Mao & Gang Li, 2023. "Bias against scientific novelty: A prepublication perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 99-114, January.
    15. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    16. Meyer, Matthias & Waldkirch, Rüdiger W. & Duscher, Irina & Just, Alexander, 2018. "Drivers of citations: An analysis of publications in “top” accounting journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 24-46.
    17. Mingkun Wei, 2020. "Research on impact evaluation of open access journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 1027-1049, February.
    18. Chunli Wei & Jingyi Zhao & Jue Ni & Jiang Li, 2023. "What does open peer review bring to scientific articles? Evidence from PLoS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 2763-2776, May.
    19. Bornmann, Lutz & Marx, Werner, 2012. "HistCite analysis of papers constituting the h index research front," Journal of Informetrics, Elsevier, vol. 6(2), pages 285-288.
    20. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:17:y:2023:i:4:s1751157723000743. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.