IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/wdkp9_v1.html
   My bibliography  Save this paper

Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading

Author

Listed:
  • Hoang, Anh-Duc

    (EdLab Asia)

Abstract

Along with discussing bibliometric analyses’ limitations and potential biases, this paper addresses the growing need for comprehensive guidelines in evaluating bibliometric research by providing systematic frameworks for both peer reviewers and readers. While numerous publications provide guidance on implementing bibliometric methods, there is a notable lack of frameworks for assessing such research, particularly regarding performance analysis and science mapping. Drawing from an extensive review of bibliometric practices and methodological literature, this paper develops structured evaluation frameworks that address the complexity of modern bibliometric analysis, introducing the VALOR framework (Verification, Alignment, Logging, Overview, Reproducibility) for assessing multi-source bibliometric studies. The paper's key contributions include comprehensive guidelines for evaluating data selection, cleaning, and analysis processes; specific criteria for assessing conceptual, intellectual, and social structure analyses; and practical guidance for integrating performance analysis with science mapping results. By providing structured frameworks for reviewers and practical guidelines for readers to interpret and apply bibliometric insights, this work enhances the rigor of bibliometric research evaluation while supporting more effective peer review processes and research planning. The paper also discusses potential areas for further development, including the integration of qualitative analysis with bibliometric data and the advancement of field-normalized metrics, ultimately aiming to support authors, reviewers, and readers in navigating the complexities of bibliometrics and enhancing the meaningfulness of bibliometric research.

Suggested Citation

  • Hoang, Anh-Duc, 2025. "Evaluating Bibliometrics Reviews: A Practical Guide for Peer Review and Critical Reading," OSF Preprints wdkp9_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:wdkp9_v1
    DOI: 10.31219/osf.io/wdkp9_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/67f4543852268ec54d829dc6/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/wdkp9_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Aria, Massimo & Cuccurullo, Corrado, 2017. "bibliometrix: An R-tool for comprehensive science mapping analysis," Journal of Informetrics, Elsevier, vol. 11(4), pages 959-975.
    2. E. C. M. Noyons & H. F. Moed & A. F. J. Raan, 1999. "Integrating research performance analysis and science mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 591-604, November.
    3. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zeng, Ying & Liu, Xinyi & Zhang, Xinyuan & Li, Zhiyong, 2024. "Retrospective of interdisciplinary research on robot services (1954–2023): From parasitism to symbiosis," Technology in Society, Elsevier, vol. 78(C).
    2. Massimo Aria & Michelangelo Misuraca & Maria Spano, 2020. "Mapping the Evolution of Social Research and Data Science on 30 Years of Social Indicators Research," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 803-831, June.
    3. Vilker Zucolotto Pessin & Luciana Harue Yamane & Renato Ribeiro Siman, 2022. "Smart bibliometrics: an integrated method of science mapping and bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3695-3718, June.
    4. Büşra Ayan & Elif Güner & Semen Son-Turan, 2022. "Blockchain Technology and Sustainability in Supply Chains and a Closer Look at Different Industries: A Mixed Method Approach," Logistics, MDPI, vol. 6(4), pages 1-39, December.
    5. Carlo Drago & Luisa Errichiello, 2024. "Remote Work admist the Covid-19 outbreak: Insights from an Ensemble Community-Based Keyword Network Analysis," Working Papers 2024.05, Fondazione Eni Enrico Mattei.
    6. Camilla Salvatore, 2023. "Inference with non-probability samples and survey data integration: a science mapping study," METRON, Springer;Sapienza Università di Roma, vol. 81(1), pages 83-107, April.
    7. Massimo Aria & Corrado Cuccurullo & Luca D’Aniello & Michelangelo Misuraca & Maria Spano, 2024. "Comparative science mapping: a novel conceptual structure analysis with metadata," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 7055-7081, November.
    8. Vincenzo Basile & Massimiliano Giacalone & Paolo Carmelo Cozzucoli, 2022. "The Impacts of Bibliometrics Measurement in the Scientific Community A Statistical Analysis of Multiple Case Studies," Review of European Studies, Canadian Center of Science and Education, vol. 14(3), pages 1-10, November.
    9. Anderson Mitterhofer Iung & Fernando Luiz Cyrino Oliveira & André Luís Marques Marcato, 2023. "A Review on Modeling Variable Renewable Energy: Complementarity and Spatial–Temporal Dependence," Energies, MDPI, vol. 16(3), pages 1-24, January.
    10. Valentina Della Corte & Giovanna Del Gaudio & Fabiana Sepe & Simone Luongo, 2021. "Destination Resilience and Innovation for Advanced Sustainable Tourism Management: A Bibliometric Analysis," Sustainability, MDPI, vol. 13(22), pages 1-19, November.
    11. Ricardo Rodrigues & Carlos Sampaio & Paulo Duarte & José Manuel Hernández-Mogollón, 2022. "Cross-Border Innovation: Assessing Concepts, Contexts, and Content," Sustainability, MDPI, vol. 14(23), pages 1-18, November.
    12. Qian Wang & Shixian Luo & Jiao Zhang & Katsunori Furuya, 2022. "Increased Attention to Smart Development in Rural Areas: A Scientometric Analysis of Smart Village Research," Land, MDPI, vol. 11(8), pages 1-28, August.
    13. Cinzia Colapinto & Issam Mejri, 2025. "The relevance of goal programming for financial portfolio management: a bibliometric and systematic literature review," Annals of Operations Research, Springer, vol. 346(2), pages 917-943, March.
    14. Cibele Oliveira Lima & Jarbas Bonetti, 2020. "Bibliometric analysis of the scientific production on coastal communities’ social vulnerability to climate change and to the impact of extreme events," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 102(3), pages 1589-1610, July.
    15. Cristina Mele & Jaqueline Pels & Maria Spano & Irene Bernardo, 2023. "Emergent understandings of the market," Italian Journal of Marketing, Springer, vol. 2023(1), pages 1-25, March.
    16. Fosso Wamba, Samuel & Bawack, Ransome Epie & Guthrie, Cameron & Queiroz, Maciel M. & Carillo, Kevin Daniel André, 2021. "Are we preparing for a good AI society? A bibliometric review and research agenda," Technological Forecasting and Social Change, Elsevier, vol. 164(C).
    17. Mismetti, Marco & Sangermano, Andrea & Del Bosco, Barbara & Bergamaschi, Mara, 2024. "Understanding the multiple facets of risk in family firms: A review of the literature and a framework for future research," Journal of Business Research, Elsevier, vol. 183(C).
    18. Ada P. Smith & Sechindra Vallury & Elizabeth Covelli Metcalf, 2023. "Social dimensions of adaptation to climate change in rangelands: a systematic literature review," Climatic Change, Springer, vol. 176(12), pages 1-24, December.
    19. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    20. Waltman, L. & van Eck, N.J.P., 2009. "A Taxonomy of Bibliometric Performance Indicators Based on the Property of Consistency," ERIM Report Series Research in Management ERS-2009-014-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:wdkp9_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.