IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v49y2025i6p1074-1102.html

Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading

Author

Listed:
  • Anh-Duc Hoang

Abstract

Along with discussing bibliometric analyses’ limitations and potential biases, this paper addresses the growing need for comprehensive guidelines in evaluating bibliometric research by providing systematic frameworks for both peer reviewers and readers. While numerous publications provide guidance on implementing bibliometric methods, there is a notable lack of frameworks for assessing such research, particularly regarding performance analysis and science mapping. Drawing from an extensive review of bibliometric practices and methodological literature, this paper develops structured evaluation frameworks that address the complexity of modern bibliometric analysis, introducing the VALOR framework (Verification, Alignment, Logging, Overview, Reproducibility) for assessing multi-source bibliometric studies. The paper’s key contributions include comprehensive guidelines for evaluating data selection, cleaning, and analysis processes; specific criteria for assessing conceptual, intellectual, and social structure analyses; and practical guidance for integrating performance analysis with science mapping results. By providing structured frameworks for reviewers and practical guidelines for readers to interpret and apply bibliometric insights, this work enhances the rigor of bibliometric research evaluation while supporting more effective peer review processes and research planning. The paper also discusses potential areas for further development, including the integration of qualitative analysis with bibliometric data and the advancement of field-normalized metrics, ultimately aiming to support authors, reviewers, and readers in navigating the complexities of bibliometrics and enhancing the meaningfulness of bibliometric research.

Suggested Citation

  • Anh-Duc Hoang, 2025. "Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading," Evaluation Review, , vol. 49(6), pages 1074-1102, December.
    • Handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:1074-1102
    DOI: 10.1177/0193841X251336839
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X251336839
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X251336839?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. E. C. M. Noyons & H. F. Moed & A. F. J. Raan, 1999. "Integrating research performance analysis and science mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 591-604, November.
    2. Aria, Massimo & Cuccurullo, Corrado, 2017. "bibliometrix: An R-tool for comprehensive science mapping analysis," Journal of Informetrics, Elsevier, vol. 11(4), pages 959-975.
    3. Bo Jarneving, 2005. "A comparison of two bibliometric methods for mapping of the research front," Scientometrics, Springer;Akadémiai Kiadó, vol. 65(2), pages 245-263, November.
    4. Li, Yunrong & Ruiz-Castillo, Javier, 2013. "The comparison of normalization procedures based on different classification systems," Journal of Informetrics, Elsevier, vol. 7(4), pages 945-958.
    5. Estevao Alves-Silva & Ana Carolina Figueira Porto & Carine Firmino & Henrique Venancio Silva & Ingrid Becker & Liegy Resende & Livia Borges & Luana Pfeffer & Marcela Silvano & Melina Santos Galdiano &, 2016. "Are the impact factor and other variables related to publishing time in ecology journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1445-1453, September.
    6. Tove Faber Frandsen & Jeppe Nicolaisen, 2008. "Intradisciplinary differences in database coverage and the consequences for bibliometric research," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(10), pages 1570-1581, August.
    7. Jingda Ding & Chao Liu & Goodluck Asobenie Kandonga, 2020. "Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1303-1322, March.
    8. Chaomei Chen, 2006. "CiteSpace II: Detecting and visualizing emerging trends and transient patterns in scientific literature," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(3), pages 359-377, February.
    9. John Hulland, 2024. "Bibliometric reviews—some guidelines," Journal of the Academy of Marketing Science, Springer, vol. 52(4), pages 935-938, July.
    10. Zhigao Liu & Yimei Yin & Weidong Liu & Michael Dunford, 2015. "Visualizing the intellectual structure and evolution of innovation systems research: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 135-158, April.
    11. Andrea Caputo & Mariya Kargina, 2022. "A user-friendly method to merge Scopus and Web of Science data during bibliometric analysis," Journal of Marketing Analytics, Palgrave Macmillan, vol. 10(1), pages 82-88, March.
    12. Oğuzhan Öztürk & Rıdvan Kocaman & Dominik K. Kanbach, 2024. "How to design bibliometric research: an overview and a framework proposal," Review of Managerial Science, Springer, vol. 18(11), pages 3333-3361, November.
    13. Henry Small, 1973. "Co‐citation in the scientific literature: A new measure of the relationship between two documents," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 24(4), pages 265-269, July.
    14. Kaur, Jasleen & Radicchi, Filippo & Menczer, Filippo, 2013. "Universality of scholarly impact metrics," Journal of Informetrics, Elsevier, vol. 7(4), pages 924-932.
    15. Ole Ellegaard & Johan A. Wallin, 2015. "The bibliometric analysis of scholarly production: How great is the impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 1809-1831, December.
    16. Tatiana Plotnikova & Bastian Rake, 2014. "Collaboration in pharmaceutical research: exploration of country-level determinants," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1173-1202, February.
    17. Vivek Kumar Singh & Prashasti Singh & Mousumi Karmakar & Jacqueline Leta & Philipp Mayr, 2021. "The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5113-5142, June.
    18. Bornmann, Lutz & Leydesdorff, Loet & Mutz, Rüdiger, 2013. "The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits," Journal of Informetrics, Elsevier, vol. 7(1), pages 158-165.
    19. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    20. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    21. Junwen Zhu & Weishu Liu, 2020. "A tale of two databases: the use of Web of Science and Scopus in academic papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 321-335, April.
    22. Philip Hallinger & Chatchai Chatpinyakoop, 2019. "A Bibliometric Review of Research on Higher Education for Sustainable Development, 1998–2018," Sustainability, MDPI, vol. 11(8), pages 1-20, April.
    23. Ludo Waltman & Nees Jan Eck, 2013. "Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 699-716, September.
    24. repec:plo:pbio00:2004089 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aldijana Bunjak & Olivier Zehnder, 2026. "Quantifying the gig economy’s evolution: A bibliometric exploration of historical patterns and future projections," Electronic Markets, Springer;IIM University of St. Gallen, vol. 36(1), pages 1-26, December.
    2. Mohd Amran Mahat & Mohd Mas Rizat Abdul Latif & Mohd Yassir Jaaffar & Mohd Tarmizi Ibrahim, 2025. "Beyond Banks and Boards: A Bibliometric Mapping of Financial Disclosure Across Islamic Institutions, 1990–2025," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 9(9), pages 5782-5793, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dušan Nikolić & Dragan Ivanović & Lidija Ivanović, 2024. "An open-source tool for merging data from multiple citation databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4573-4595, July.
    2. Zhao Zhi-hong & Wei Zi-yi & Yao Shan-ji, 2025. "How Entrepreneurship University Enhances Digital Opportunities? Evidences from Bibliometric and Topic Modeling," SAGE Open, , vol. 15(3), pages 21582440251, August.
    3. Massimo Aria & Michelangelo Misuraca & Maria Spano, 2020. "Mapping the Evolution of Social Research and Data Science on 30 Years of Social Indicators Research," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 803-831, June.
    4. Büşra Ayan & Elif Güner & Semen Son-Turan, 2022. "Blockchain Technology and Sustainability in Supply Chains and a Closer Look at Different Industries: A Mixed Method Approach," Logistics, MDPI, vol. 6(4), pages 1-39, December.
    5. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    6. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    7. Ricardo Rodrigues & Carlos Sampaio & Paulo Duarte & José Manuel Hernández-Mogollón, 2022. "Cross-Border Innovation: Assessing Concepts, Contexts, and Content," Sustainability, MDPI, vol. 14(23), pages 1-18, November.
    8. Qian Wang & Shixian Luo & Jiao Zhang & Katsunori Furuya, 2022. "Increased Attention to Smart Development in Rural Areas: A Scientometric Analysis of Smart Village Research," Land, MDPI, vol. 11(8), pages 1-28, August.
    9. Song Xu & Huichen Gao, 2025. "Navigating Research Frontiers in China’s Rural Planning: A Bibliometric Analysis of Sustainable Development," Sustainability, MDPI, vol. 17(1), pages 1-23, January.
    10. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    11. Zeng, Ying & Liu, Xinyi & Zhang, Xinyuan & Li, Zhiyong, 2024. "Retrospective of interdisciplinary research on robot services (1954–2023): From parasitism to symbiosis," Technology in Society, Elsevier, vol. 78(C).
    12. Dima Jamali & Georges Samara & Lokman I. Meho, 2024. "Determinants of research productivity and efficiency among the Arab world’s accredited business schools," Management Review Quarterly, Springer, vol. 74(4), pages 2511-2543, December.
    13. Charlie Karlsson & Björn Hammarfelt, 2025. "Correction: The growth and development of Nordic regional science research 1982–2022: bibliometric evidence from thirteen regional science journals," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 74(2), pages 1-1, June.
    14. Souzanchi Kashani, Ebrahim & Roshani, Saeed, 2019. "Evolution of innovation system literature: Intellectual bases and emerging trends," Technological Forecasting and Social Change, Elsevier, vol. 146(C), pages 68-80.
    15. Cihangir Kaçmaz & Osman Tayyar Çelik, 2025. "Life Satisfaction in Adolescence: A Bibliometric Analysis," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 18(5), pages 1927-1955, October.
    16. Zhichao Wang & Valentin Zelenyuk, 2021. "Performance Analysis of Hospitals in Australia and its Peers: A Systematic Review," CEPA Working Papers Series WP012021, School of Economics, University of Queensland, Australia.
    17. Shuo Xu & Liyuan Hao & Xin An & Hongshen Pang & Ting Li, 2020. "Review on emerging research topics with key-route main path analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 607-624, January.
    18. Manta Eduard Mihai & Davidescu Adriana Ana Maria & Geambasu Maria Cristina & Florescu Margareta Stela, 2023. "Exploring the research area of direct taxation. An empirical analysis based on bibliometric analysis results," Management & Marketing, Sciendo, vol. 18(s1), pages 355-383, December.
    19. Chengliang Liu & Qinchang Gui, 2016. "Mapping intellectual structures and dynamics of transport geography research: a scientometric overview from 1982 to 2014," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(1), pages 159-184, October.
    20. Md. Nazmus Sakib & Shah Ridwan Chowdhury & Mohammad Younus & Nehad Laila Sanju & Farhana Foysal Satata & Mahafuza Islam, 2024. "How HR analytics evolved over time: a bibliometric analysis on Scopus database," Future Business Journal, Springer, vol. 10(1), pages 1-22, December.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:49:y:2025:i:6:p:1074-1102. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.