IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i3p590-597.html
   My bibliography  Save this article

Revisiting the scientometric conceptualization of impact and its measurement

Author

Listed:
  • Abramo, Giovanni

Abstract

The development of scientometric indicators and methods for evaluative purposes, requires a multitude of assumptions, conventions, limitations, and caveats. Given this, we cannot permit ambiguities in the key concepts forming the basis of scientometric science itself, or research assessment exercises would rest on quicksand. This conceptual work attempts to spell out some principles leading to a clear definition of “impact” of research, and above all, of the appropriate scientometric indicator to measure it. The aim is to stimulate a discussion aimed at a definitive convergence on the meaning and measurement of a fundamental concept of the scientometric science.

Suggested Citation

  • Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:3:p:590-597
    DOI: 10.1016/j.joi.2018.05.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157718301925
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.05.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    2. Bornmann, Lutz & Leydesdorff, Loet & Wang, Jian, 2014. "How to improve the prediction based on citation impact percentiles for years shortly after the publication date?," Journal of Informetrics, Elsevier, vol. 8(1), pages 175-180.
    3. Stegehuis, Clara & Litvak, Nelly & Waltman, Ludo, 2015. "Predicting the long-term citation impact of recent publications," Journal of Informetrics, Elsevier, vol. 9(3), pages 642-657.
    4. Terrence A. Brooks, 1986. "Evidence of complex citer motivations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 37(1), pages 34-36, January.
    5. Bronwyn H. Hall & Adam Jaffe & Manuel Trajtenberg, 2005. "Market Value and Patent Citations," RAND Journal of Economics, The RAND Corporation, vol. 36(1), pages 16-38, Spring.
    6. M. H. MacRoberts & B. R. MacRoberts, 1987. "Another test of the normative theory of citing," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 38(4), pages 305-306, July.
    7. M. H. MacRoberts & B. R. MacRoberts, 1988. "Author motivation for not citing influences: A methodological note," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 39(6), pages 432-433, November.
    8. Terrence A. Brooks, 1985. "Private acts and public objects: An investigation of citer motivations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 36(4), pages 223-229, July.
    9. Michael H. MacRoberts & Barbara R. MacRoberts, 2018. "The mismeasure of science: Citation analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(3), pages 474-482, March.
    10. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    11. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2014. "How do you define and measure research productivity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1129-1144, November.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    13. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    14. Mike Thelwall, 2017. "Are Mendeley reader counts useful impact indicators in all fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1721-1731, December.
    15. Cassidy R. Sugimoto & Sam Work & Vincent Larivière & Stefanie Haustein, 2017. "Scholarly use of social media and altmetrics: A review of the literature," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2037-2062, September.
    16. Gibbons, Michael & Johnston, Ron, 1974. "The roles of science in technological innovation," Research Policy, Elsevier, vol. 3(3), pages 220-242, November.
    17. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    18. Jonathan Adams, 2005. "Early citation counts correlate with accumulated impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(3), pages 567-581, June.
    19. Emanuela Reale & Anna Barbara & Antonio Costantini, 2006. "Peer review for the evaluation of the academic research: the Italian experience," CERIS Working Paper 200615, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    20. Tahamtan, Iman & Bornmann, Lutz, 2018. "Core elements in the process of citing publications: Conceptual overview of the literature," Journal of Informetrics, Elsevier, vol. 12(1), pages 203-216.
    21. J Mingers, 2008. "Exploring the dynamics of journal citations: Modelling with s-curves," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(8), pages 1013-1025, August.
    22. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    23. Jian Wang, 2013. "Citation time window choice for research impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 851-872, March.
    24. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2016. "A farewell to the MNCS and like size-independent indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 646-651.
    25. Michael H. MacRoberts & Barbara R. MacRoberts, 1989. "Problems of citation analysis: A critical review," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 40(5), pages 342-349, September.
    26. Lokman I. Meho & Diane H. Sonnenwald, 2000. "Citation ranking versus peer evaluation of senior faculty research performance: A case study of Kurdish scholarship," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(2), pages 123-138.
    27. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: a comparison of peer review and bibliometrics rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 929-941, December.
    28. Norman Kaplan, 1965. "The norms of citation behavior: Prolegomena to the footnote," American Documentation, Wiley Blackwell, vol. 16(3), pages 179-184, July.
    29. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    30. Wolfgang Glänzel & Balázs Schlemmer & Bart Thijs, 2003. "Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(3), pages 571-586, November.
    31. David I Stern, 2014. "High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-11, November.
    32. Susanne E. Baumgartner & Loet Leydesdorff, 2014. "Group-based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims”," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 797-811, April.
    33. Lutz Bornmann, 2014. "Is there currently a scientific revolution in Scientometrics?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(3), pages 647-648, March.
    34. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2010. "Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 821-833, September.
    35. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    36. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    37. Rodrigo Costas & Zohreh Zahedi & Paul Wouters, 2015. "Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 2003-2019, October.
    38. Alberto Anfossi & Alberto Ciolfi & Filippo Costa & Giorgio Parisi & Sergio Benedetto, 2016. "Large-scale assessment of research outputs through a weighted combination of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 671-683, May.
    39. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    40. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2021. "On the relation between the degree of internationalization of cited and citing publications: A field level analysis, including and excluding self-citations," Journal of Informetrics, Elsevier, vol. 15(1).
    2. Ruth Zárate-Rueda & Yolima Ivonne Beltrán-Villamizar & Daniella Murallas-Sánchez, 2021. "Social representations of socioenvironmental dynamics in extractive ecosystems and conservation practices with sustainable development: a bibliometric analysis," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 23(11), pages 16428-16453, November.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2019. "The collaboration behavior of top scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 215-232, January.
    4. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2021. "A bibliometric methodology to unveil territorial inequities in the scientific wealth to combat COVID-19," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6601-6624, August.
    5. Guoliang Lyu & Ganwei Shi, 2019. "On an approach to boosting a journal’s citation potential," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1387-1409, September.
    6. Aslıhan Sezgin & Keziban Orbay & Metin Orbay, 2022. "Educational Research Review From Diverse Perspectives: A Bibliometric Analysis of Web of Science (2011–2020)," SAGE Open, , vol. 12(4), pages 21582440221, December.
    7. Giovanni Abramo & Francesca Apponi & Ciriaco Andrea D'Angelo, 2021. "Public–private research collaborations: Longitudinal field‐level analysis of determinants, frequency, and impact," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1405-1427, December.
    8. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Carloni, Massimiliano, 2019. "The balance of knowledge flows," Journal of Informetrics, Elsevier, vol. 13(1), pages 1-9.
    9. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2020. "A novel methodology to assess the scientific standing of nations at field level," Journal of Informetrics, Elsevier, vol. 14(1).
    10. Wolfgang G. Stock & Isabelle Dorsch & Gerhard Reichmann & Christian Schlögl, 2023. "Labor productivity, labor impact, and co-authorship of research institutions: publications and citations per full-time equivalents," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 363-377, January.
    11. Emilio Abad-Segura & Alfonso Infante-Moro & Mariana-Daniela González-Zamar & Eloy López-Meneses, 2021. "Blockchain Technology for Secure Accounting Management: Research Trends Analysis," Mathematics, MDPI, vol. 9(14), pages 1-26, July.
    12. Yu, Dejian & Sheng, Libo, 2021. "Influence difference main path analysis: Evidence from DNA and blockchain domain citation networks," Journal of Informetrics, Elsevier, vol. 15(4).
    13. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    14. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    15. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Felici, Giovanni, 2019. "Predicting publication long-term impact through a combination of early citations and journal impact factor," Journal of Informetrics, Elsevier, vol. 13(1), pages 32-49.
    16. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    17. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2020. "The domestic localization of knowledge flows as evidenced by publication citation: the case of Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1305-1329, November.
    18. Yang, Siluo & Zheng, Mengxue & Yu, Yonghao & Wolfram, Dietmar, 2021. "Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?," Journal of Informetrics, Elsevier, vol. 15(1).
    19. Tanmoy Konar, 2021. "Author-Suggested, Weighted Citation Index: A Novel Approach for Determining the Contribution of Individual Researchers," Publications, MDPI, vol. 9(3), pages 1-8, July.
    20. Giovanni Abramo & Francesca Apponi & Ciriaco Andrea D'Angelo, 2021. "Public-private research collaborations: longitudinal field-level analysis of determinants, frequency and impact," Papers 2103.14857, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Felici, Giovanni, 2019. "Predicting publication long-term impact through a combination of early citations and journal impact factor," Journal of Informetrics, Elsevier, vol. 13(1), pages 32-49.
    2. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    3. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    4. Thelwall, Mike & Nevill, Tamara, 2018. "Could scientists use Altmetric.com scores to predict longer term citation counts?," Journal of Informetrics, Elsevier, vol. 12(1), pages 237-248.
    5. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    6. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    7. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    8. Martorell Cunil, Onofre & Otero González, Luis & Durán Santomil, Pablo & Mulet Forteza, Carlos, 2023. "How to accomplish a highly cited paper in the tourism, leisure and hospitality field," Journal of Business Research, Elsevier, vol. 157(C).
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2021. "A bibliometric methodology to unveil territorial inequities in the scientific wealth to combat COVID-19," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6601-6624, August.
    10. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    11. Wang, Xing & Zhang, Zhihui, 2020. "Improving the reliability of short-term citation impact indicators by taking into account the correlation between short- and long-term citation impact," Journal of Informetrics, Elsevier, vol. 14(2).
    12. Guoliang Lyu & Ganwei Shi, 2019. "On an approach to boosting a journal’s citation potential," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1387-1409, September.
    13. Abramo, Giovanni & Aksnes, Dag W. & D’Angelo, Ciriaco Andrea, 2020. "Comparison of research performance of Italian and Norwegian professors and universities," Journal of Informetrics, Elsevier, vol. 14(2).
    14. Mike Thelwall, 2018. "Early Mendeley readers correlate with later citation counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1231-1240, June.
    15. David L. Anderson & John Tressler, 2016. "Citation-Capture Rates for Economics Journals: Do they Differ from Other Disciplines and Does it Matter?," Economic Papers, The Economic Society of Australia, vol. 35(1), pages 73-85, March.
    16. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    17. Stegehuis, Clara & Litvak, Nelly & Waltman, Ludo, 2015. "Predicting the long-term citation impact of recent publications," Journal of Informetrics, Elsevier, vol. 9(3), pages 642-657.
    18. Schreiber, Michael, 2015. "Restricting the h-index to a publication and citation time window: A case study of a timed Hirsch index," Journal of Informetrics, Elsevier, vol. 9(1), pages 150-155.
    19. Hou, Jianhua & Yang, Xiucai, 2020. "Social media-based sleeping beauties: Defining, identifying and features," Journal of Informetrics, Elsevier, vol. 14(2).
    20. Brito, Ricardo & Rodríguez-Navarro, Alonso, 2018. "Research assessment by percentile-based double rank analysis," Journal of Informetrics, Elsevier, vol. 12(1), pages 315-329.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:3:p:590-597. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.