IDEAS home Printed from
   My bibliography  Save this article

The ability of different peer review procedures to flag problematic publications


  • S. P. J. M. Horbach

    () (Radboud University
    Leiden University)

  • W. Halffman

    () (Radboud University)


There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review’s ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

Suggested Citation

  • S. P. J. M. Horbach & W. Halffman, 2019. "The ability of different peer review procedures to flag problematic publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 339-373, January.
  • Handle: RePEc:spr:scient:v:118:y:2019:i:1:d:10.1007_s11192-018-2969-2
    DOI: 10.1007/s11192-018-2969-2

    Download full text from publisher

    File URL:
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Ludo Waltman & Nees Jan van Eck, 2012. "A new methodology for constructing a publication‐level classification system of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    2. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    3. Azoulay, Pierre & Bonatti, Alessandro & Krieger, Joshua L., 2017. "The career effects of scandal: Evidence from scientific retractions," Research Policy, Elsevier, vol. 46(9), pages 1552-1569.
    4. Brandli Stitzel & Gary A. Hoover & William Clark, 2018. "More on Plagiarism in the Social Sciences," Social Science Quarterly, Southwestern Social Science Association, vol. 99(3), pages 1075-1088, September.
    5. Hopp, Christian & Hoover, Gary A., 2017. "How prevalent is academic misconduct in management research?," Journal of Business Research, Elsevier, vol. 80(C), pages 73-81.
    6. Tianwei He, 2013. "Retraction of global scientific publications from 2001 to 2010," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 555-561, August.
    7. Marion Schmidt, 2018. "An analysis of the validity of retraction annotation in pubmed and the web of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(2), pages 318-328, February.
    8. Yýldýrým Beyazýt ÇÝÇEN, 2017. "II. International Strategic Research Congress," Journal of Economics and Political Economy, KSP Journals, vol. 4(4), pages 423-424, December.
    Full references (including those not matched with items on IDEAS)


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:118:y:2019:i:1:d:10.1007_s11192-018-2969-2. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Springer Nature Abstracting and Indexing). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.