IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v14y2020i3s1751157719303682.html
   My bibliography  Save this article

Bibliometrics and systematic reviews: A comparison between the Proknow-C and the Methodi Ordinatio

Author

Listed:
  • de Carvalho, Gustavo Dambiski Gomes
  • Sokulski, Carla Cristiane
  • da Silva, Wesley Vieira
  • de Carvalho, Hélio Gomes
  • de Moura, Rafael Vignoli
  • de Francisco, Antonio Carlos
  • da Veiga, Claudimar Pereira

Abstract

This research aims to analyze and compare the selection of relevant papers by two bibliometric methods, namely, the Knowledge Development Process-Constructivist (Proknow-C) and the Methodi Ordinatio, which were also discussed in terms of common steps, limitations and underlying assumptions. Particularly, the topic of knowledge management in small and medium-sized enterprises was chosen in order to illustrate the application of these two methods that support systematic reviews. Besides descriptive indexes such as citations and journal impact, co-citation analysis was employed to examine selected papers. The results showed that both methods possess a high agreement rate in terms of selecting recent articles, but a low agreement rate regarding relevant/seminal articles, besides revealing a strong bias of the Methodi Ordinatio towards recent articles when the alpha constant is 10. Particularly, recent papers (up to two years of publication) represent about 80 % of top-50 InOrdinatio ranked articles. The co-citation analysis of the 46 selected articles (29 recent; 17 relevant) corroborated the methods capacity to select relevant articles. Finally, further bibliometric reviews in other topics may benefit from the common steps, limitations and underlying assumptions discussed in this research for selecting and analyzing relevant papers.

Suggested Citation

  • de Carvalho, Gustavo Dambiski Gomes & Sokulski, Carla Cristiane & da Silva, Wesley Vieira & de Carvalho, Hélio Gomes & de Moura, Rafael Vignoli & de Francisco, Antonio Carlos & da Veiga, Claudimar Per, 2020. "Bibliometrics and systematic reviews: A comparison between the Proknow-C and the Methodi Ordinatio," Journal of Informetrics, Elsevier, vol. 14(3).
    • Handle: RePEc:eee:infome:v:14:y:2020:i:3:s1751157719303682
    DOI: 10.1016/j.joi.2020.101043
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157719303682
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2020.101043?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Martin-Martin, Alberto & Orduna-Malea, Enrique & Harzing, Anne-Wil & Delgado López-Cózar, Emilio, 2017. "Can we use Google Scholar to identify highly-cited documents?," Journal of Informetrics, Elsevier, vol. 11(1), pages 152-163.
    2. Stephen J. Bensman, 2012. "The impact factor: its place in Garfield’s thought, in science evaluation, and in library collection management," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 263-275, August.
    3. Houqiang Yu, 2017. "Context of altmetrics data matters: an investigation of count type and user category," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 267-283, April.
    4. Lutz Bornmann, 2014. "How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature," Research Evaluation, Oxford University Press, vol. 23(2), pages 166-173.
    5. David I. Stern, 2017. "Comment on Bornmann (2017): confidence intervals for journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1811-1813, December.
    6. Michel Zitt, 2012. "The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay’s article 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 485-503, August.
    7. Sandra Mara Iesbik Valmorbida & Sandra Rolim Ensslin, 2017. "Performance evaluation of university rankings : literature review and guidelines for future research," International Journal of Business Innovation and Research, Inderscience Enterprises Ltd, vol. 14(4), pages 479-501.
    8. Lutz Bornmann & Robin Haunschild, 2017. "Quality and impact considerations in bibliometrics: a reply to Ricker (in press)," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1857-1859, June.
    9. Fang Liu & Guangyuan Hu & Li Tang & Weishu Liu, 2018. "The penalty of containing more non-English articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 359-366, January.
    10. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    11. Dorta-González, P. & Dorta-González, M.I., 2013. "Impact maturity times and citation time windows: The 2-year maximum journal impact factor," Journal of Informetrics, Elsevier, vol. 7(3), pages 593-602.
    12. Anne-Wil Harzing & Satu Alakangas, 2016. "Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 787-804, February.
    13. Juan Miguel Campanario, 2018. "Are leaders really leading? Journals that are first in Web of Science subject categories in the context of their groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 111-130, April.
    14. Nees Jan Eck & Ludo Waltman, 2017. "Citation-based clustering of publications using CitNetExplorer and VOSviewer," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 1053-1070, May.
    15. Christine Meschede & Tobias Siebenlist, 2018. "Cross-metric compatability and inconsistencies of altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 283-297, April.
    16. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    17. Regina Negri Pagani & João Luiz Kovaleski & Luis Mauricio Resende, 2015. "Methodi Ordinatio: a proposed methodology to select and rank relevant scientific papers encompassing the impact factor, number of citation, and year of publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2109-2135, December.
    18. Pardeep Sud & Mike Thelwall, 2014. "Evaluating altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1131-1143, February.
    19. Nees Jan Eck & Ludo Waltman, 2010. "Software survey: VOSviewer, a computer program for bibliometric mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(2), pages 523-538, August.
    20. Zohreh Zahedi & Rodrigo Costas & Paul Wouters, 2014. "How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1491-1513, November.
    21. Anne-Wil Harzing & Satu Alakangas, 2017. "Microsoft Academic is one year old: the Phoenix is ready to leave the nest," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1887-1894, September.
    22. Pablo Diniz Batista & Igor Marques-Carneiro & Leduc Hermeto de Almeida Fauth & Márcia de Oliveira Reis Brandão, 2018. "Web of Science: Showing a Bug Today That Can Mislead Scientific Research Output Prediction," SAGE Open, , vol. 8(1), pages 21582440187, February.
    23. Binglu Wang & Yi Bu & Yang Xu, 2018. "A quantitative exploration on reasons for citing articles from the perspective of cited authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 675-687, August.
    24. Jian Wang, 2013. "Citation time window choice for research impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 851-872, March.
    25. Gualberto Buela-Casal & Izabela Zych & Ana Medina & María I. Viedma Del Jesus & Susana Lozano & Gloria Torres, 2009. "Analysis of the influence of the two types of the journal articles; theoretical and empirical on the impact factor of a journal," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(1), pages 265-282, July.
    26. Emilio Delgado López-Cózar & Nicolás Robinson-García & Daniel Torres-Salinas, 2014. "The Google scholar experiment: How to index false papers and manipulate bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(3), pages 446-454, March.
    27. Martin Ricker, 2017. "Letter to the Editor: About the quality and impact of scientific articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1851-1855, June.
    28. Mark R. Elkins & Christopher G. Maher & Robert D. Herbert & Anne M. Moseley & Catherine Sherrington, 2010. "Correlation between the Journal Impact Factor and three other journal citation indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 81-93, October.
    29. Elisha R. T. Chiware & Deborah Becker, 2018. "Citation Patterns of Conference Proceedings in Master’s and Doctoral Studies: A Case Study of Information Technology and Systems," SAGE Open, , vol. 8(2), pages 21582440187, April.
    30. Éric Archambault & Vincent Larivière, 2009. "History of the journal impact factor: Contingencies and consequences," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(3), pages 635-649, June.
    31. Anne‐Wil Harzing & Ron van der Wal, 2009. "A Google Scholar h‐index for journals: An alternative metric to measure journal impact in economics and business," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 41-46, January.
    32. Gualberto Buela-Casal & Izabela Zych, 2012. "What do the scientists think about the impact factor?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 281-292, August.
    33. Donald O. Case & Georgeann M. Higgins, 2000. "How can we investigate citation behavior? A study of reasons for citing literature in communication," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(7), pages 635-645.
    34. Mauricio Johnny Loos & Eugenio Merino & Carlos Manuel Taboada Rodriguez, 2016. "Mapping the state of the art of ergonomics within logistics," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(1), pages 85-101, October.
    35. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    36. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    37. Jerome K. Vanclay, 2009. "Bias in the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(1), pages 3-12, January.
    38. Lutz Bornmann, 2015. "Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 1123-1144, June.
    39. Elaine Aparecida Regiani Campos & Regina Negri Pagani & Luis Mauricio Resende & Joseane Pontes, 2018. "Construction and qualitative assessment of a bibliographic portfolio using the methodology Methodi Ordinatio," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 815-842, August.
    40. Rodrigo Costas & Zohreh Zahedi & Paul Wouters, 2015. "Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 2003-2019, October.
    41. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    42. Lutz Bornmann, 2017. "Confidence intervals for Journal Impact Factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1869-1871, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vivek Warke & Satish Kumar & Arunkumar Bongale & Ketan Kotecha, 2021. "Sustainable Development of Smart Manufacturing Driven by the Digital Twin Framework: A Statistical Analysis," Sustainability, MDPI, vol. 13(18), pages 1-49, September.
    2. Mauricio A. Correa-Ochoa & Roxana Bedoya & Luisa M. Gómez & David Aguiar & Carlos A. Palacio-Tobón & Henry A. Colorado, 2023. "A Review on the Characterization and Measurement of the Carbonaceous Fraction of Particulate Matter," Sustainability, MDPI, vol. 15(11), pages 1-23, May.
    3. Khalid Ahmed Al-Ansari & Ahmet Faruk Aysan, 2021. "More than ten years of Blockchain creation: How did we use the technology and which direction is the research heading? [Plus de dix ans de création Blockchain : Comment avons-nous utilisé la techno," Working Papers hal-03343048, HAL.
    4. Vilker Zucolotto Pessin & Luciana Harue Yamane & Renato Ribeiro Siman, 2022. "Smart bibliometrics: an integrated method of science mapping and bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3695-3718, June.
    5. Gustavo Dambiski Gomes de Carvalho & Luis Mauricio Martins de Resende & Joseane Pontes & Hélio Gomes de Carvalho & Leozenir Mendes Betim, 2021. "Innovation and Management in MSMEs: A Literature Review of Highly Cited Papers," SAGE Open, , vol. 11(4), pages 21582440211, October.
    6. Helton Rogger Regatieri & Oswaldo Hideo Ando Junior & José Ricardo Cezar Salgado, 2022. "Systematic Review of Lithium-Ion Battery Recycling Literature Using ProKnow-C and Methodi Ordinatio," Energies, MDPI, vol. 15(4), pages 1-23, February.
    7. Marco Antonio Cruz-Morato & Josefa García-Mestanza & Carmen Dueñas-Zambrana, 2021. "Special Employment Centres, Time Factor and Sustainable Human Resources Management in Spanish Hotel Industry: Can Corporate Social Marketing Improve the Labour Situation of People with Disabilities?," Sustainability, MDPI, vol. 13(19), pages 1-24, September.
    8. Pech, Gerson & Delgado, Catarina, 2021. "Screening the most highly cited papers in longitudinal bibliometric studies and systematic literature reviews of a research field or journal: Widespread used metrics vs a percentile citation-based app," Journal of Informetrics, Elsevier, vol. 15(3).
    9. Jorge Alfredo Cerqueira-Streit & Gustavo Yuho Endo & Patricia Guarnieri & Luciano Batista, 2021. "Sustainable Supply Chain Management in the Route for a Circular Economy: An Integrative Literature Review," Logistics, MDPI, vol. 5(4), pages 1-21, November.
    10. Mauricio A. Correa-Ochoa & Juliana Rojas & Luisa M. Gómez & David Aguiar & Carlos A. Palacio-Tobón & Henry A. Colorado, 2023. "Systematic Search Using the Proknow-C Method for the Characterization of Atmospheric Particulate Matter Using the Materials Science Techniques XRD, FTIR, XRF, and Raman Spectroscopy," Sustainability, MDPI, vol. 15(11), pages 1-23, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    3. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    4. Mike Thelwall, 2018. "Does Microsoft Academic find early citations?," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 325-334, January.
    5. Michael Thelwall, 2018. "Can Microsoft Academic be used for citation analysis of preprint archives? The case of the Social Science Research Network," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 913-928, May.
    6. Mingyang Wang & Shijia Jiao & Kah-Hin Chai & Guangsheng Chen, 2019. "Building journal’s long-term impact: using indicators detected from the sustained active articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 261-283, October.
    7. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    8. Thelwall, Mike, 2018. "Dimensions: A competitor to Scopus and the Web of Science?," Journal of Informetrics, Elsevier, vol. 12(2), pages 430-435.
    9. Yaxue Ma & Zhichao Ba & Yuxiang Zhao & Jin Mao & Gang Li, 2021. "Understanding and predicting the dissemination of scientific papers on social media: a two-step simultaneous equation modeling–artificial neural network approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7051-7085, August.
    10. Mike Thelwall, 2017. "Are Mendeley reader counts useful impact indicators in all fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1721-1731, December.
    11. Thelwall, Mike, 2018. "Microsoft Academic automatic document searches: Accuracy for journal articles and suitability for citation analysis," Journal of Informetrics, Elsevier, vol. 12(1), pages 1-9.
    12. Lutz Bornmann & Robin Haunschild & Vanash M Patel, 2020. "Are papers addressing certain diseases perceived where these diseases are prevalent? The proposal to use Twitter data as social-spatial sensors," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-22, November.
    13. Martín-Martín, Alberto & Orduna-Malea, Enrique & Delgado López-Cózar, Emilio, 2018. "Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community," Journal of Informetrics, Elsevier, vol. 12(2), pages 494-509.
    14. Blandy Pamplona Solis & Julio César Cruz Argüello & Leopoldo Gómez Barba & Mayra Polett Gurrola & Zakaryaa Zarhri & Danna Lizeth TrejoArroyo, 2019. "Bibliometric Analysis of the Mass Transport in a Gas Diffusion Layer in PEM Fuel Cells," Sustainability, MDPI, vol. 11(23), pages 1-18, November.
    15. Gustavo Dambiski Gomes de Carvalho & Luis Mauricio Martins de Resende & Joseane Pontes & Hélio Gomes de Carvalho & Leozenir Mendes Betim, 2021. "Innovation and Management in MSMEs: A Literature Review of Highly Cited Papers," SAGE Open, , vol. 11(4), pages 21582440211, October.
    16. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    17. Latefa Ali Dardas & Malik Sallam & Amanda Woodward & Nadia Sweis & Narjes Sweis & Faleh A. Sawair, 2023. "Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive," Publications, MDPI, vol. 11(1), pages 1-16, January.
    18. Saeed-Ul Hassan & Mubashir Imran & Uzair Gillani & Naif Radi Aljohani & Timothy D. Bowman & Fereshteh Didegah, 2017. "Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1037-1057, November.
    19. Derek R. Smith, 2012. "Impact factors, scientometrics and the history of citation-based research," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 419-427, August.
    20. Sergio Copiello, 2020. "The alleged citation advantage of video abstracts may be a matter of self-citations and self-selection bias. Comment on “The impact of video abstract on citation counts” by Zong et al," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 751-757, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:14:y:2020:i:3:s1751157719303682. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.