IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v129y2024i3d10.1007_s11192-024-04935-2.html
   My bibliography  Save this article

RCE (rationale–cogency–extent) criterion unravels features affecting citation impact of top-ranked systematic literature reviews: leaving the impression…is all you need

Author

Listed:
  • Marko Orošnjak

    (University of Novi Sad)

  • Branko Štrbac

    (University of Novi Sad)

  • Srđan Vulanović

    (University of Novi Sad)

  • Biserka Runje

    (University of Zagreb)

  • Amalija Horvatić Novak

    (University of Zagreb)

  • Andrej Razumić

    (University of Zagreb)

Abstract

Hijacked from medical and health sciences, Systematic Literature Reviews (SLRs) are widely (ab)used in many scientific domains. Considering the ability to provide transparency and replicability of research results, many scientists consider an SLR a safe avenue for attaining scientific impact, given that the theoretical probability of acceptance is relatively high. Relying on dual analysis of Partial Least Squares Discriminant Analysis (PLS-DA) and Network Analysis (NA), the study identifies key features associated with citation impact within top-tier SLRs. Next, the study introduces Rationale, Cogency, and Extent (RCE) criterion for evaluating potential markers that predict citation impact using two case studies of SLRs from engineering domain. The findings suggest that the informal logic for starting a review significantly correlates with citation impact. Additionally, journal- and author-level metrics, along with RCE composite scores, display significant difference between top- and bottom-ranked SLRs. Through NA, reporting the quality assessment of studies (QATR) emerges as the most influential node within the RCE network. Despite its lack of direct correlation with citation impact, we conclude that QATR is a moderating variable. Finally, the study concludes that a well-articulated research question, alignment with existing evidence, and rigorous data use collectively serve as a blueprint for producing a high-quality SLR.

Suggested Citation

  • Marko Orošnjak & Branko Štrbac & Srđan Vulanović & Biserka Runje & Amalija Horvatić Novak & Andrej Razumić, 2024. "RCE (rationale–cogency–extent) criterion unravels features affecting citation impact of top-ranked systematic literature reviews: leaving the impression…is all you need," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(3), pages 1891-1947, March.
    • Handle: RePEc:spr:scient:v:129:y:2024:i:3:d:10.1007_s11192-024-04935-2
    DOI: 10.1007/s11192-024-04935-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-024-04935-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-024-04935-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wagner, Gerit & Prester, Julian & Roche, Maria & Schryen, Guido & Benlian, Alexander & Paré, Guy & Templier, Mathieu, 2021. "Which factors affect the scientific impact of review papers in IS research? A scientometric study," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 125255, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    2. Juan Xie & Kaile Gong & Ying Cheng & Qing Ke, 2019. "The correlation between paper length and citations: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 763-786, March.
    3. Andreas Nishikawa-Pacher, 2022. "Research Questions with PICO: A Universal Mnemonic," Publications, MDPI, vol. 10(3), pages 1-10, June.
    4. Guy Paré & Mary Tate & David Johnstone & Spyros Kitsiou, 2016. "Contextualizing the twin concepts of systematicity and transparency in information systems literature reviews," European Journal of Information Systems, Taylor & Francis Journals, vol. 25(6), pages 493-508, November.
    5. Mingyang Wang & Guang Yu & Daren Yu, 2011. "Mining typical features for highly cited papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 695-706, June.
    6. Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    7. Mingyang Wang & Guang Yu & Shuang An & Daren Yu, 2012. "Discovery of factors influencing citation impact based on a soft fuzzy rough set model," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 635-644, December.
    8. Olalekan A Uthman & Charles I Okwundu & Charles S Wiysonge & Taryn Young & Aileen Clarke, 2013. "Citation Classics in Systematic Reviews and Meta-Analyses: Who Wrote the Top 100 Most Cited Articles?," PLOS ONE, Public Library of Science, vol. 8(10), pages 1-1, October.
    9. Rousseau, Sandra & Catalano, Giuseppe & Daraio, Cinzia, 2021. "Can we estimate a monetary value of scientific publications?," Research Policy, Elsevier, vol. 50(1).
    10. Bornmann, Lutz & Leydesdorff, Loet, 2017. "Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data," Journal of Informetrics, Elsevier, vol. 11(1), pages 164-175.
    11. Wagner, Gerit & Prester, Julian & Roche, Maria & Benlian, Alexander & Schryen, Guido, 2016. "Factors Affecting the Scientific Impact of Literature Reviews: A Scientometric Study," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 83356, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    12. Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
    13. Minho So & Jiyoung Kim & Sangki Choi & Han Park, 2015. "Factors affecting citation networks in science and technology: focused on non-quality factors," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(4), pages 1513-1530, July.
    14. Danlu Liu & Jiaxin Jin & Jinhui Tian & Kehu Yang, 2015. "Quality Assessment and Factor Analysis of Systematic Reviews and Meta-Analyses of Endoscopic Ultrasound Diagnosis," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-13, April.
    15. Juan Xie & Kaile Gong & Jiang Li & Qing Ke & Hyonchol Kang & Ying Cheng, 2019. "A probe into 66 factors which are possibly associated with the number of citations an article received," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1429-1454, June.
    16. Mathieu Templier & Guy Paré, 2018. "Transparency in literature reviews: an assessment of reporting practices across review types and genres in top IS journals," European Journal of Information Systems, Taylor & Francis Journals, vol. 27(5), pages 503-550, September.
    17. Tian Yu & Guang Yu & Peng-Yu Li & Liang Wang, 2014. "Citation impact prediction for scientific papers using stepwise regression analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1233-1252, November.
    18. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    19. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    20. Vanclay, Jerome K., 2013. "Factors affecting citation rates in environmental science," Journal of Informetrics, Elsevier, vol. 7(2), pages 265-271.
    21. Wolfgang Glänzel & Koenraad Debackere & Bart Thijs & András Schubert, 2006. "A concise review on the role of author self-citations in information science, bibliometrics and science policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(2), pages 263-277, May.
    22. John S. Liu & Chung-Huei Kuan, 2016. "A new approach for main path analysis: Decay in knowledge diffusion," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(2), pages 465-476, February.
    23. Mingyang Wang & Zhenyu Wang & Guangsheng Chen, 2019. "Which can better predict the future success of articles? Bibliometric indices or alternative metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1575-1595, June.
    24. T. Liskiewicz & G. Liskiewicz & J. Paczesny, 2021. "Factors affecting the citations of papers in tribology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3321-3336, April.
    25. Mei Hsiu-Ching Ho & John S. Liu & Kerr C.-T. Chang, 2017. "To include or not: the role of review papers in citation-based analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 65-76, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kayvan Kousha & Mike Thelwall, 2024. "Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(3), pages 215-244, March.
    2. Martorell Cunil, Onofre & Otero González, Luis & Durán Santomil, Pablo & Mulet Forteza, Carlos, 2023. "How to accomplish a highly cited paper in the tourism, leisure and hospitality field," Journal of Business Research, Elsevier, vol. 157(C).
    3. Ruan, Xuanmin & Zhu, Yuanyang & Li, Jiang & Cheng, Ying, 2020. "Predicting the citation counts of individual papers via a BP neural network," Journal of Informetrics, Elsevier, vol. 14(3).
    4. Zhang, Xinyuan & Xie, Qing & Song, Min, 2021. "Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network," Journal of Informetrics, Elsevier, vol. 15(2).
    5. Mingyang Wang & Zhenyu Wang & Guangsheng Chen, 2019. "Which can better predict the future success of articles? Bibliometric indices or alternative metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1575-1595, June.
    6. Lutz Bornmann & Adam Y. Ye & Fred Y. Ye, 2018. "Identifying “hot papers” and papers with “delayed recognition” in large-scale datasets by using dynamically normalized citation impact scores," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 655-674, August.
    7. Li, Xin & Ma, Xiaodi & Feng, Ye, 2024. "Early identification of breakthrough research from sleeping beauties using machine learning," Journal of Informetrics, Elsevier, vol. 18(2).
    8. Juan Xie & Kaile Gong & Ying Cheng & Qing Ke, 2019. "The correlation between paper length and citations: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 763-786, March.
    9. Ha, Taehyun, 2022. "An explainable artificial-intelligence-based approach to investigating factors that influence the citation of papers," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    10. Wanjun Xia & Tianrui Li & Chongshou Li, 2023. "A review of scientific impact prediction: tasks, features and methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 543-585, January.
    11. Nunkoo, Robin & Hall, C. Michael & Rughoobur-Seetah, Soujata & Teeroovengadum, Viraiyan, 2019. "Citation practices in tourism research: Toward a gender conscientious engagement," Annals of Tourism Research, Elsevier, vol. 79(C).
    12. Kofi A. A-O. Agyei-Henaku & Charlotte Badu-Prah & Francis Srofenyoh & Ferguson K. Gidiglo & Akua Agyeiwaa-Afrane & Justice G. Djokoto, 2024. "Citations of Publications on Foreign Direct Investments into Agribusiness: Nature, Variability and Drivers," SAGE Open, , vol. 14(1), pages 21582440241, February.
    13. Linda C. Smith, 2024. "Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(3), pages 245-267, March.
    14. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    15. Liu, Qiuling & Guo, Lei & Sun, Yiping & Ren, Linlin & Wang, Xinhua & Han, Xiaohui, 2024. "Do scholars' collaborative tendencies impact the quality of their publications? A generalized propensity score matching analysis," Journal of Informetrics, Elsevier, vol. 18(1).
    16. Guoqiang Liang & Haiyan Hou & Xiaodan Lou & Zhigang Hu, 2019. "Qualifying threshold of “take-off” stage for successfully disseminated creative ideas," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1193-1208, September.
    17. Giuseppe Pernagallo, 2023. "Science in the mist: A model of asymmetric information for the research market," Metroeconomica, Wiley Blackwell, vol. 74(2), pages 390-415, May.
    18. Giovanni Abramo & Ciriaco Andrea D’Angelo & Leonardo Grilli, 2024. "The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 5003-5019, August.
    19. Kehan Wang & Wenxuan Shi & Junsong Bai & Xiaoping Zhao & Liying Zhang, 2021. "Prediction and application of article potential citations based on nonlinear citation-forecasting combined model," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6533-6550, August.
    20. Yadav, Pratyush & Pervin, Nargis, 2022. "Towards efficient navigation in digital libraries: Leveraging popularity, semantics and communities to recommend scholarly articles," Journal of Informetrics, Elsevier, vol. 16(4).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:129:y:2024:i:3:d:10.1007_s11192-024-04935-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.