IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i12d10.1007_s11192-021-04199-0.html
   My bibliography  Save this article

A comparison of systematic reviews and guideline-based systematic reviews in medical studies

Author

Listed:
  • Alexander Schniedermann

    (German Centre for Higher Education Research and Science Studies)

Abstract

The question of how citation impact relates to academic quality accompanies every decade in bibliometric research. Although experts have employed more complex conceptions of research quality for responsible evaluation, detailed analyses of how impact relates to dimensions such as methodological rigor are lacking. But the increasing number of formal guidelines for biomedical research offer not only the potential to understand the social dynamics of standardization, but also their relations to scientific rewards. By using data from Web of Science and PubMed, this study focuses on systematic reviews from biomedicine and compares this genre with those systematic reviews that applied the PRISMA reporting standard. Besides providing an overview about growth and location, it was found that the latter, more standardized type of systematic review accumulates more citations. It is argued that instead of reinforcing the traditional conception that higher impact represents higher quality, highly prolific authors could be more inclined to develop and apply new standards than more average researchers. In addition, research evaluation would benefit from a more nuanced conception of scientific output which respects the intellectual role of various document types.

Suggested Citation

  • Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:12:d:10.1007_s11192-021-04199-0
    DOI: 10.1007/s11192-021-04199-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-04199-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-04199-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christian Lachance & Steve Poirier & Vincent Larivière, 2014. "The kiss of death? The effect of being cited in a review on subsequent citations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(7), pages 1501-1505, July.
    2. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "Evaluating research: from informed peer review to bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 499-514, June.
    4. Tiiu Ojasoo & Hervé Maisonneuve & Jean-Christophe Doré, 2001. "Evaluating publication trends in clinical research: How reliable are medical databases?," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(3), pages 391-404, March.
    5. Barrios, Maite & Guilera, Georgina & Gómez-Benito, Juana, 2013. "Impact and structural features of meta-analytical studies, standard articles and reviews in psychology: Similarities and differences," Journal of Informetrics, Elsevier, vol. 7(2), pages 478-486.
    6. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    7. Christian B van der Pol & Matthew D F McInnes & William Petrcich & Adam S Tunis & Ramez Hanna, 2015. "Is Quality and Completeness of Reporting of Systematic Reviews and Meta-Analyses Published in High Impact Radiology Journals Associated with Citation Rates?," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-15, March.
    8. Berkwits, Michael, 1998. "From practice to research: The case for criticism in an age of evidence," Social Science & Medicine, Elsevier, vol. 47(10), pages 1539-1545, November.
    9. Moreira, Tiago, 2005. "Diversity in clinical guidelines: the role of repertoires of evaluation," Social Science & Medicine, Elsevier, vol. 60(9), pages 1975-1985, May.
    10. Lutz Bornmann, 2013. "How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(3), pages 587-595, March.
    11. Robert Colebunders & Chris Kenyon & Ronald Rousseau, 2014. "Increase in numbers and proportions of review articles in Tropical Medicine, Infectious Diseases, and oncology," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(1), pages 201-205, January.
    12. David Moher & Lex Bouter & Sabine Kleinert & Paul Glasziou & Mai Har Sham & Virginia Barbour & Anne-Marie Coriat & Nicole Foeger & Ulrich Dirnagl, 2020. "The Hong Kong Principles for assessing researchers: Fostering research integrity," PLOS Biology, Public Library of Science, vol. 18(7), pages 1-14, July.
    13. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    14. Jian Wang, 2013. "Citation time window choice for research impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 851-872, March.
    15. Jefferson Seide Molléri & Kai Petersen & Emilia Mendes, 2018. "Towards understanding the relation between citations and research quality in software engineering studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1453-1478, December.
    16. David A Groneberg & Stefan Rolle & Michael H K Bendels & Doris Klingelhöfer & Norman Schöffel & Jan Bauer & Dörthe Brüggmann, 2019. "A world map of evidence-based medicine: Density equalizing mapping of the Cochrane database of systematic reviews," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-10, December.
    17. Paul T E Cusack, 2020. "On Pain," Biomedical Journal of Scientific & Technical Research, Biomedical Research Network+, LLC, vol. 31(3), pages 24253-24254, October.
    18. Ruth Müller & Sarah de Rijcke, 2017. "Exploring the epistemic impacts of academic performance indicators in the life sciences," Research Evaluation, Oxford University Press, vol. 26(3), pages 157-168.
    19. Jochen Gläser & Grit Laudel, 2001. "Integrating Scientometric Indicators into Sociological Studies: Methodical and Methodological Problems," Scientometrics, Springer;Akadémiai Kiadó, vol. 52(3), pages 411-434, November.
    20. Paul Donner, 2017. "Document type assignment accuracy in the journal citation index data of Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 219-236, October.
    21. Hilda Bastian & Paul Glasziou & Iain Chalmers, 2010. "Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?," PLOS Medicine, Public Library of Science, vol. 7(9), pages 1-6, September.
    22. Anne-Wil Harzing, 2013. "Document categories in the ISI Web of Knowledge: Misunderstanding the Social Sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 23-34, January.
    23. Matthew J Page & Larissa Shamseer & Douglas G Altman & Jennifer Tetzlaff & Margaret Sampson & Andrea C Tricco & Ferrán Catalá-López & Lun Li & Emma K Reid & Rafael Sarkis-Onofre & David Moher, 2016. "Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study," PLOS Medicine, Public Library of Science, vol. 13(5), pages 1-30, May.
    24. Ruth Müller & Sarah de Rijcke, 2017. "Thinking with Indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences," Research Evaluation, Oxford University Press, vol. 26(4), pages 361-361.
    25. Ronald Rousseau, 2012. "Basic properties of both percentile rank scores and the I3 indicator," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(2), pages 416-420, February.
    26. Ronald Rousseau, 2012. "Basic properties of both percentile rank scores and the I3 indicator," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(2), pages 416-420, February.
    27. Li, Kai & Yan, Erjia & Feng, Yuanyuan, 2017. "How is R cited in research outputs? Structure, impacts, and citation standard," Journal of Informetrics, Elsevier, vol. 11(4), pages 989-1002.
    28. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    29. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    30. Lutz Bornmann & Richard Williams, 2020. "An evaluation of percentile measures of citation impact, and a proposal for making them better," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1457-1478, August.
    31. Stephen, Dimity & Stahlschmidt, Stephan, 2021. "Performance and structures of the German science system 2021," Studien zum deutschen Innovationssystem 5-2021, Expertenkommission Forschung und Innovation (EFI) - Commission of Experts for Research and Innovation, Berlin.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    2. Miranda, Ruben & Garcia-Carpintero, Esther, 2018. "Overcitation and overrepresentation of review papers in the most cited papers," Journal of Informetrics, Elsevier, vol. 12(4), pages 1015-1030.
    3. Bornmann, Lutz & Leydesdorff, Loet & Wang, Jian, 2013. "Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P1," Journal of Informetrics, Elsevier, vol. 7(4), pages 933-944.
    4. Bornmann, Lutz, 2013. "The problem of citation impact assessments for recent publication years in institutional evaluations," Journal of Informetrics, Elsevier, vol. 7(3), pages 722-729.
    5. Michaela Strinzel & Josh Brown & Wolfgang Kaltenbrunner & Sarah Rijcke & Michael Hill, 2021. "Ten ways to improve academic CVs for fairer research assessment," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    6. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    7. Mei Hsiu-Ching Ho & John S. Liu & Kerr C.-T. Chang, 2017. "To include or not: the role of review papers in citation-based analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 65-76, January.
    8. Gerson Pech & Catarina Delgado, 2020. "Assessing the publication impact using citation data from both Scopus and WoS databases: an approach validated in 15 research fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 909-924, November.
    9. Paul Donner, 2017. "Document type assignment accuracy in the journal citation index data of Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 219-236, October.
    10. Gabriel-Alexandru Vîiu & Mihai Păunescu, 2021. "The citation impact of articles from which authors gained monetary rewards based on journal metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4941-4974, June.
    11. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    12. Brito, Ricardo & Rodríguez-Navarro, Alonso, 2018. "Research assessment by percentile-based double rank analysis," Journal of Informetrics, Elsevier, vol. 12(1), pages 315-329.
    13. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
    14. Laura Sheble, 2017. "Macro‐level diffusion of a methodological knowledge innovation: Research synthesis methods, 1972–2011," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(12), pages 2693-2708, December.
    15. de Carvalho, Gustavo Dambiski Gomes & Sokulski, Carla Cristiane & da Silva, Wesley Vieira & de Carvalho, Hélio Gomes & de Moura, Rafael Vignoli & de Francisco, Antonio Carlos & da Veiga, Claudimar Per, 2020. "Bibliometrics and systematic reviews: A comparison between the Proknow-C and the Methodi Ordinatio," Journal of Informetrics, Elsevier, vol. 14(3).
    16. Neal R. Haddaway & Matthew J. Page & Chris C. Pritchard & Luke A. McGuinness, 2022. "PRISMA2020: An R package and Shiny app for producing PRISMA 2020‐compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(2), June.
    17. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile-based bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    18. Mike Thelwall, 2021. "Alternative medicines worth researching? Citation analyses of acupuncture, chiropractic, homeopathy, and osteopathy 1996–2017," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8731-8747, October.
    19. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    20. Boshen Jiao & Anirban Basu & Joshua Roth & M. Bender & Ilsa Rovira & Traci Clemons & Dalyna Quach & Scott Ramsey & Beth Devine, 2021. "The Use of Cost-Effectiveness Analysis in Sickle Cell Disease: A Critical Review of the Literature," PharmacoEconomics, Springer, vol. 39(11), pages 1225-1241, November.

    More about this item

    Keywords

    Systematic reviews; Reporting guidelines; PRISMA; Standardization; Document types;
    All these keywords.

    JEL classification:

    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:12:d:10.1007_s11192-021-04199-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.