IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i1d10.1007_s11192-020-03545-y.html
   My bibliography  Save this article

Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ

Author

Listed:
  • Qianjin Zong

    (South China Normal University)

  • Yafen Xie

    (Guangdong Provincial Work Injury Rehabilitation Hospital)

  • Jiechun Liang

    (South China Normal University)

Abstract

This study aims to investigate whether open peer review can improve citation count. Articles published in PeerJ during 2013 and 2015 were chosen as the data set. Two categories of the articles were generated: articles with closed peer review history and articles with open peer review history. A propensity score matching with the radius matching method was performed using 14 confounding variables. The other five common matching methods of propensity score matching, the bias-adjusted matching estimator developed by Abadie and Imbens (Simple and bias-corrected matching estimators for average treatment effects, National Bureau of Economic Research, Cambridge, pp 1–57, 2002), and the data set excluding articles with an extremely high citation count were used to check the robustness of the results. The results of this study showed that articles with open peer review history could be expected to have significantly greater citation counts than articles with closed peer review history. Our results suggest that open peer review can improve citation count, and that the best practice for open peer review might be a compromise open peer review.

Suggested Citation

  • Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
  • Handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03545-y
    DOI: 10.1007/s11192-020-03545-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03545-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-020-03545-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Philip M. Davis & Michael J. Fromerth, 2007. "Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 203-215, May.
    2. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    3. Feng Guo & Chao Ma & Qingling Shi & Qingqing Zong, 2018. "Succinct effect or informative effect: the relationship between title length and the number of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1531-1539, September.
    4. Lutz Bornmann & Markus Wolf & Hans-Dieter Daniel, 2012. "Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 843-856, June.
    5. Matthias Gnewuch & Klaus Wohlrabe, 2017. "Title characteristics and citations in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1573-1578, March.
    6. Alberto Falk Delgado & Gregory Garretson & Anna Falk Delgado, 2019. "The language of peer review reports on articles published in the BMJ, 2014–2017: an observational study," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1225-1235, September.
    7. Jelte M Wicherts, 2016. "Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-19, January.
    8. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    9. Jeffrey T Leek & Margaret A Taub & Fernando J Pineda, 2011. "Cooperation between Referees and Authors Increases Peer Review Accuracy," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-11, November.
    10. Uddin, Shahadat & Khan, Arif, 2016. "The impact of author-selected keywords on citation counts," Journal of Informetrics, Elsevier, vol. 10(4), pages 1166-1177.
    11. Afshin Sadeghi & Sarven Capadisli & Johannes Wilm & Christoph Lange & Philipp Mayr, 2019. "Opening and Reusing Transparent Peer Reviews with Automatic Article Annotation," Publications, MDPI, Open Access Journal, vol. 7(1), pages 1-12, February.
    12. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    13. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    14. Fereshteh Didegah & Timothy D. Bowman & Kim Holmberg, 2018. "On the differences between citations and altmetrics: An investigation of factors driving altmetrics versus citations for finnish articles," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(6), pages 832-843, June.
    15. Lutz Bornmann & Werner Marx & Hermann Schier & Andreas Thor & Hans-Dieter Daniel, 2010. "From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics," Research Evaluation, Oxford University Press, vol. 19(2), pages 105-118, June.
    16. Tony Ross-Hellauer & Arvid Deppe & Birgit Schmidt, 2017. "Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers," PLOS ONE, Public Library of Science, vol. 12(12), pages 1-28, December.
    17. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    18. Syavash Nobarany & Kellogg S. Booth, 2015. "Use of politeness strategies in signed open peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(5), pages 1048-1064, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Eirini Delikoura & Dimitrios Kouis, 2021. "Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece," Publications, MDPI, Open Access Journal, vol. 9(2), pages 1-19, March.
    2. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 0. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 0, pages 1-19.
    2. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    3. Kong, Ling & Wang, Dongbo, 2020. "Comparison of citations and attention of cover and non-cover papers," Journal of Informetrics, Elsevier, vol. 14(4).
    4. Qianjin Zong & Yafen Xie & Rongchan Tuo & Jingshi Huang & Yang Yang, 2019. "The impact of video abstract on citation counts: evidence from a retrospective cohort study of New Journal of Physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1715-1727, June.
    5. Rajeev Dehejia, 2013. "The Porous Dialectic: Experimental and Non-Experimental Methods in Development Economics," WIDER Working Paper Series wp-2013-011, World Institute for Development Economic Research (UNU-WIDER).
    6. Apps, Patricia & Mendolia, Silvia & Walker, Ian, 2013. "The impact of pre-school on adolescents’ outcomes: Evidence from a recent English cohort," Economics of Education Review, Elsevier, vol. 37(C), pages 183-199.
    7. Tymon Słoczyński, 2015. "The Oaxaca–Blinder Unexplained Component as a Treatment Effects Estimator," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 77(4), pages 588-604, August.
    8. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    9. Fatema, Naureen, 2019. "Can land title reduce low-intensity interhousehold conflict incidences and associated damages in eastern DRC?," World Development, Elsevier, vol. 123(C), pages 1-1.
    10. Cadot, Olivier & Fernandes, Ana M. & Gourdon, Julien & Mattoo, Aaditya, 2015. "Are the benefits of export support durable? Evidence from Tunisia," Journal of International Economics, Elsevier, vol. 97(2), pages 310-324.
    11. von Greiff, Jenny, 2009. "Displacement and self-employment entry," Labour Economics, Elsevier, vol. 16(5), pages 556-565, October.
    12. Caballero, Julián, 2021. "Corporate dollar debt and depreciations: All’s well that ends well?," Journal of Banking & Finance, Elsevier, vol. 130(C).
    13. Chen, Xinxin & Shi, Yaojiang & Rozelle, Scott, 2007. "Does Taking One Step Back Get You Two Steps Forward? Grade Retention and School Performance in Rural China," MPRA Paper 10917, University Library of Munich, Germany.
    14. Jasmin Kantarevic & Boris Kralj, 2016. "Physician Payment Contracts in the Presence of Moral Hazard and Adverse Selection: The Theory and Its Application in Ontario," Health Economics, John Wiley & Sons, Ltd., vol. 25(10), pages 1326-1340, October.
    15. Huber, Martin & Lechner, Michael & Walter, Thomas & Wunsch, Conny, 2009. "Do German Welfare-to-Work Programmes Reduce Welfare and Increase Work?," CEPR Discussion Papers 7238, C.E.P.R. Discussion Papers.
    16. David McKenzie & John Gibson & Steven Stillman, 2010. "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, MIT Press, vol. 8(4), pages 913-945, June.
    17. Andreas Peichl & Nico Pestel & Sebastian Siegloch, 2013. "The politicians’ wage gap: insights from German members of parliament," Public Choice, Springer, vol. 156(3), pages 653-676, September.
    18. Mano, Yukichi & Akoten, John & Yoshino, Yutaka & Sonobe, Tetsushi, 2014. "Teaching KAIZEN to small business owners: An experiment in a metalworking cluster in Nairobi," Journal of the Japanese and International Economies, Elsevier, vol. 33(C), pages 25-42.
    19. Martin Huber & Michael Lechner & Giovanni Mellace, 2017. "Why Do Tougher Caseworkers Increase Employment? The Role of Program Assignment as a Causal Mechanism," The Review of Economics and Statistics, MIT Press, vol. 99(1), pages 180-183, March.
    20. Shen, Chung-Hua & Wu, Meng-Wen & Chen, Ting-Hsuan & Fang, Hao, 2016. "To engage or not to engage in corporate social responsibility: Empirical evidence from global banking sector," Economic Modelling, Elsevier, vol. 55(C), pages 207-225.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03545-y. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.