IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v124y2020i3d10.1007_s11192-020-03548-9.html
   My bibliography  Save this article

On bibliometrics in academic promotions: a case study in computer science and engineering in Italy

Author

Listed:
  • Camil Demetrescu

    (Sapienza University of Rome)

  • Irene Finocchi

    (Sapienza University of Rome)

  • Andrea Ribichini

    (Sapienza University of Rome)

  • Marco Schaerf

    (Sapienza University of Rome)

Abstract

Due to its quantitative nature, bibliometrics is becoming increasingly popular among policy makers for academic hiring and career promotions. In this article, we quantitatively assess the impact that the granularity level in the classification of scientific areas would entail on research evaluation based on bibliometric indicators. We use as a case study the Italian national habilitation system (ASN), which classifies faculty members according to their academic discipline and relies on journal counts, citations, and h-indices as a basis for promoting tenure track researchers to associate professors and associate to full professors. The assessment checks whether the individual indicators of a researcher are above a certain threshold, e.g., the median over the population of researchers working in the same discipline. Our investigation focuses on two related, rather broad disciplines: computer science and computer engineering. We show that the ASN practice of using the same thresholds for all members of a scientific discipline can favor certain sub-communities that are characterized by higher bibliometric indicators, and disfavor others. We report evidence that up to 30% of Italian faculty members of certain sub-communities would see their indicators drop below the threshold, thus becoming not eligible for promotion, if the ASN were conducted on a more accurate, fine-grained classification. Conversely, in the same scenario, up to 11% of faculty members, in different sub-communities, would see their indicators rise above the threshold, granting them eligibility. Our data set includes 1685 authors, 89,185 distinct publications, and 262,286 author-publication pairs.

Suggested Citation

  • Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2020. "On bibliometrics in academic promotions: a case study in computer science and engineering in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2207-2228, September.
    • Handle: RePEc:spr:scient:v:124:y:2020:i:3:d:10.1007_s11192-020-03548-9
    DOI: 10.1007/s11192-020-03548-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03548-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03548-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    2. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2014. "How do you define and measure research productivity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1129-1144, November.
    3. Ivan Simko, 2015. "Analysis of bibliometric indicators to determine citation bias," Palgrave Communications, Palgrave Macmillan, vol. 1(palcomms2), pages 15011-15011, June.
    4. Marzolla, Moreno, 2015. "Quantitative analysis of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 9(2), pages 285-316.
    5. Marco Pautasso, 2015. "The Italian University Habilitation and the Challenge of Increasing the Representation of Women in Academia," Challenges, MDPI, vol. 6(1), pages 1-16, March.
    6. Juan Gorraiz & Christian Gumpenberger & Thomas Glade, 2016. "On the bibliometric coordinates of four different research fields in Geography," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 873-897, May.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2014. "Career advancement and scientific performance in universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 891-907, February.
    8. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2016. "A farewell to the MNCS and like size-independent indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 646-651.
    9. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    10. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    11. Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2018. "Accuracy of author names in bibliographic data sources: an Italian case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1777-1791, December.
    12. Marzolla, Moreno, 2016. "Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 10(2), pages 408-438.
    13. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    14. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    15. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2022. "On computer science research and its temporal evolution," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4913-4938, August.
    2. Matthias Kuppler, 2022. "Predicting the future impact of Computer Science researchers: Is there a gender bias?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6695-6732, November.
    3. Ghassan Abdul-Majeed & Wissam Mahmood & Nasri S. M. Namer, 2021. "Measuring research performance of Iraqi universities using Scopus data," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2349-2363, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    2. Giovanni Abramo & Les Oxley, 2021. "Scientometric‐based analysis in business and economics: Introduction, examples, and guidelines," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1261-1270, December.
    3. Marzolla, Moreno, 2016. "Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 10(2), pages 408-438.
    4. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    5. Zhenbin Yan & Qiang Wu & Xingchen Li, 2016. "Do Hirsch-type indices behave the same in assessing single publications? An empirical study of 29 bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1815-1833, December.
    6. Perc, Matjaž, 2010. "Zipf’s law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia’s research as an example," Journal of Informetrics, Elsevier, vol. 4(3), pages 358-364.
    7. Maria Cristiana Martini & Elvira Pelle & Francesco Poggi & Andrea Sciandra, 2022. "The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 5633-5659, October.
    8. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    9. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    10. Amodio, Pierluigi & Brugnano, Luigi & Scarselli, Filippo, 2021. "Implementation of the PaperRank and AuthorRank indices in the Scopus database," Journal of Informetrics, Elsevier, vol. 15(4).
    11. Giovanni Abramo & Ciriaco D’Angelo, 2015. "An assessment of the first “scientific habilitation” for university appointments in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 32(3), pages 329-357, December.
    12. Brandão, Luana Carneiro & Soares de Mello, João Carlos Correia Baptista, 2019. "A multi-criteria approach to the h-index," European Journal of Operational Research, Elsevier, vol. 276(1), pages 357-363.
    13. Andersen, Jens Peter, 2017. "An empirical and theoretical critique of the Euclidean index," Journal of Informetrics, Elsevier, vol. 11(2), pages 455-465.
    14. Sidiropoulos, A. & Gogoglou, A. & Katsaros, D. & Manolopoulos, Y., 2016. "Gazing at the skyline for star scientists," Journal of Informetrics, Elsevier, vol. 10(3), pages 789-813.
    15. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    16. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    17. Jingda Ding & Chao Liu & Goodluck Asobenie Kandonga, 2020. "Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1303-1322, March.
    18. Yves Fassin, 2020. "The HF-rating as a universal complement to the h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 965-990, November.
    19. Deming Lin & Tianhui Gong & Wenbin Liu & Martin Meyer, 2020. "An entropy-based measure for the evolution of h index research," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2283-2298, December.
    20. Sten F Odenwald, 2020. "A citation study of earth science projects in citizen science," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-26, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:124:y:2020:i:3:d:10.1007_s11192-020-03548-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.