IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v10y2016i3p854-862.html
   My bibliography  Save this article

From rankings to funnel plots: The question of accounting for uncertainty when assessing university research performance

Author

Listed:
  • Abramo, Giovanni
  • D’Angelo, Andrea Ciriaco
  • Grilli, Leonardo

Abstract

The work applies the funnel plot methodology to measure and visualize uncertainty in the research performance of Italian universities in the science disciplines. The performance assessment is carried out at both discipline and overall university level. The findings reveal that for most universities the citation-based indicator used gives insufficient statistical evidence to infer that their research productivity is inferior or superior to the average. This general observation is one that we could indeed expect in a higher education system that is essentially non-competitive. The question is whether the introduction of uncertainty in performance reporting, while technically sound, could weaken institutional motivation to work towards continuous improvement.

Suggested Citation

  • Abramo, Giovanni & D’Angelo, Andrea Ciriaco & Grilli, Leonardo, 2016. "From rankings to funnel plots: The question of accounting for uncertainty when assessing university research performance," Journal of Informetrics, Elsevier, vol. 10(3), pages 854-862.
    • Handle: RePEc:eee:infome:v:10:y:2016:i:3:p:854-862
    DOI: 10.1016/j.joi.2016.07.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157716300736
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2016.07.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems," Journal of Informetrics, Elsevier, vol. 6(2), pages 155-168.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Grilli, Leonardo, 2015. "Funnel plots for visualizing uncertainty in the research performance of institutions," Journal of Informetrics, Elsevier, vol. 9(4), pages 954-961.
    3. Christopher Claassen, 2015. "Measuring university quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 793-807, September.
    4. Ciriaco Andrea D'Angelo & Cristiano Giuffrida & Giovanni Abramo, 2011. "A heuristic approach to author name disambiguation in bibliometrics databases for large‐scale research assessments," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 257-269, February.
    5. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S. & van Raan, Anthony F.J., 2011. "Towards a new crown indicator: Some theoretical considerations," Journal of Informetrics, Elsevier, vol. 5(1), pages 37-47.
    6. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    7. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    8. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2016. "A farewell to the MNCS and like size-independent indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 646-651.
    9. Ciriaco Andrea D'Angelo & Cristiano Giuffrida & Giovanni Abramo, 2011. "A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(2), pages 257-269, February.
    10. Francesca Ieva & Anna Paganoni, 2015. "Detecting and visualizing outliers in provider profiling via funnel plots and mixed effect models," Health Care Management Science, Springer, vol. 18(2), pages 166-172, June.
    11. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    12. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Rosati, Francesco, 2013. "The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences," Journal of Informetrics, Elsevier, vol. 7(1), pages 198-208.
    13. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2023. "A heuristic approach based on Leiden rankings to identify outliers: evidence from Italian universities in the European landscape," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 483-510, January.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Soldatenkova, Anastasiia, 2016. "The ratio of top scientists to the academic staff as an indicator of the competitive strength of universities," Journal of Informetrics, Elsevier, vol. 10(2), pages 596-605.
    3. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Grilli, Leonardo, 2015. "Funnel plots for visualizing uncertainty in the research performance of institutions," Journal of Informetrics, Elsevier, vol. 9(4), pages 954-961.
    2. Jan Schulz, 2016. "Using Monte Carlo simulations to assess the impact of author name disambiguation quality on different bibliometric analyses," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1283-1298, June.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Anastasiia Soldatenkova, 2017. "How long do top scientists maintain their stardom? An analysis by region, gender and discipline: evidence from Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 867-877, February.
    4. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    5. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Evaluating university research: Same performance indicator, different rankings," Journal of Informetrics, Elsevier, vol. 9(3), pages 514-525.
    6. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2014. "Are the authors of highly cited articles also the most productive ones?," Journal of Informetrics, Elsevier, vol. 8(1), pages 89-97.
    7. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    8. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2016. "A comparison of university performance scores and ranks by MNCS and FSS," Journal of Informetrics, Elsevier, vol. 10(4), pages 889-901.
    9. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2013. "The impact of unproductive and top researchers on overall university research performance," Journal of Informetrics, Elsevier, vol. 7(1), pages 166-175.
    10. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Soldatenkova, Anastasiia, 2017. "An investigation on the skewness patterns and fractal nature of research productivity distributions at field and discipline level," Journal of Informetrics, Elsevier, vol. 11(1), pages 324-335.
    11. Thelwall, Mike, 2016. "Are there too many uncited articles? Zero inflated variants of the discretised lognormal and hooked power law distributions," Journal of Informetrics, Elsevier, vol. 10(2), pages 622-633.
    12. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Soldatenkova, Anastasiia, 2016. "The ratio of top scientists to the academic staff as an indicator of the competitive strength of universities," Journal of Informetrics, Elsevier, vol. 10(2), pages 596-605.
    13. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "Diversification versus specialization in scientific research: Which strategy pays off?," Technovation, Elsevier, vol. 82, pages 51-57.
    14. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    15. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2014. "Assessing national strengths and weaknesses in research fields," Journal of Informetrics, Elsevier, vol. 8(3), pages 766-775.
    16. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2015. "Should the research performance of scientists be distinguished by gender?," Journal of Informetrics, Elsevier, vol. 9(1), pages 25-38.
    17. Giovanni Abramo & Ciriaco D’Angelo, 2015. "An assessment of the first “scientific habilitation” for university appointments in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 32(3), pages 329-357, December.
    18. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2014. "Career advancement and scientific performance in universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 891-907, February.
    19. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    20. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2016. "The north–south divide in the Italian higher education system," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2093-2117, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:10:y:2016:i:3:p:854-862. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.