IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v115y2018i1d10.1007_s11192-018-2666-1.html
   My bibliography  Save this article

University rankings: What do they really show?

Author

Listed:
  • Jill Johnes

    (University of Huddersfield)

Abstract

University rankings as developed by the media are used by many stakeholders in higher education: students looking for university places; academics looking for university jobs; university managers who need to maintain standing in the competitive arena of student recruitment; and governments who want to know that public funds spent on universities are delivering a world class higher education system. Media rankings deliberately draw attention to the performance of each university relative to all others, and as such they are undeniably simple to use and interpret. But one danger is that they are potentially open to manipulation and gaming because many of the measures underlying the rankings are under the control of the institutions themselves. This paper examines media rankings (constructed from an amalgamation of variables representing performance across numerous dimensions) to reveal the problems with using a composite index to reflect overall performance. It ends with a proposal for an alternative methodology which leads to groupings rather than point estimates.

Suggested Citation

  • Jill Johnes, 2018. "University rankings: What do they really show?," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 585-606, April.
    • Handle: RePEc:spr:scient:v:115:y:2018:i:1:d:10.1007_s11192-018-2666-1
    DOI: 10.1007/s11192-018-2666-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-018-2666-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-018-2666-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Muller, Seán M, 2017. "Academics as rent seekers: distorted incentives in higher education, with reference to the South African case," International Journal of Educational Development, Elsevier, vol. 52(C), pages 58-67.
    2. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    3. Popov, Sergey V. & Bernhardt, Dan, 2010. "University Competition, Grading Standards and Grade Inflation," MPRA Paper 26461, University Library of Munich, Germany.
    4. Johnes, Jill, 1996. "Performance assessment in higher education in Britain," European Journal of Operational Research, Elsevier, vol. 89(1), pages 18-33, February.
    5. Johnes, Geraint, 1992. "Performance Indicators in Higher Education: A Survey of Recent Work," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 8(2), pages 19-34, Summer.
    6. Kelchtermans, Stijn & Verboven, Frank, 2010. "Program duplication in higher education is not necessarily bad," Journal of Public Economics, Elsevier, vol. 94(5-6), pages 397-409, June.
    7. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    8. Ronald G. Ehrenberg, 2012. "American Higher Education in Transition," Journal of Economic Perspectives, American Economic Association, vol. 26(1), pages 193-216, Winter.
    9. Saisana, Michaela & d'Hombres, Béatrice & Saltelli, Andrea, 2011. "Rickety numbers: Volatility of university rankings and policy implications," Research Policy, Elsevier, vol. 40(1), pages 165-177, February.
    10. Sergey V. Popov & Dan Bernhardt, 2013. "University Competition, Grading Standards, And Grade Inflation," Economic Inquiry, Western Economic Association International, vol. 51(3), pages 1764-1778, July.
    11. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    12. De Fraja, Gianni & Valbonesi, Paola, 2012. "The design of the university system," Journal of Public Economics, Elsevier, vol. 96(3), pages 317-330.
    13. Michela Nardo & Michaela Saisana & Andrea Saltelli & Stefano Tarantola & Anders Hoffman & Enrico Giovannini, 2005. "Handbook on Constructing Composite Indicators: Methodology and User Guide," OECD Statistics Working Papers 2005/3, OECD Publishing.
    14. Marie-Laure Bougnol & José Dulá, 2006. "Validating DEA as a ranking tool: An application of DEA to assess performance in higher education," Annals of Operations Research, Springer, vol. 145(1), pages 339-365, July.
    15. Mika Nieminen, 2009. "Influence of research funding and science policy on university research performance: A comparison of five countries," Science and Public Policy, Oxford University Press, vol. 36(6), pages 419-430, July.
    16. Laurens Cherchye & Willem Moesen & Nicky Rogge & Tom Puyenbroeck, 2007. "An Introduction to ‘Benefit of the Doubt’ Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 82(1), pages 111-145, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christoph Burmann & Fernando García & Francisco Guijarro & Javier Oliver, 2021. "Ranking the Performance of Universities: The Role of Sustainability," Sustainability, MDPI, vol. 13(23), pages 1-16, November.
    2. Jill Johnes & Swati Virmani, 2020. "Chief executive pay in UK higher education: the role of university performance," Annals of Operations Research, Springer, vol. 288(2), pages 547-576, May.
    3. Lutz Bornmann & Werner Marx, 2020. "Thomas theorem in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 553-555, April.
    4. Ernesto Galbán-Rodríguez & Deborah Torres-Ponjuán & Ricardo Arencibia-Jorge, 2021. "Multidimensional quantitative analysis of the Cuban scientific output and its regional context," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2643-2665, March.
    5. Yating Wen & Xiaodong Zhao & Xingguo Li & Yuqi Zang, 2023. "Explaining the Paradox of World University Rankings in China: Higher Education Sustainability Analysis with Sentiment Analysis and LDA Topic Modeling," Sustainability, MDPI, vol. 15(6), pages 1-18, March.
    6. McAleer, M.J. & Nakamura, T. & Watkins, C., 2018. "Size, Internationalization and University Rankings: Evaluating Times Higher Education (THE) Data for Japan," Econometric Institute Research Papers EI2018-43, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    7. Juan Antonio Dip, 2021. "What does U-multirank tell us about knowledge transfer and research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3011-3039, April.
    8. Csóka, Imola & Sebestyén, Géza & Neszveda, Gábor, 2019. "Tudományos teljesítmény mérése a magyar felsőoktatás gazdasági képzéseiben [Measuring scientific performance of business and economics faculties in Hungarian higher education]," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(7), pages 751-770.
    9. Fernando García & Francisco Guijarro & Javier Oliver, 2021. "A Multicriteria Goal Programming Model for Ranking Universities," Mathematics, MDPI, vol. 9(5), pages 1-17, February.
    10. Michael McAleer & Tamotsu Nakamura & Clinton Watkins, 2019. "Size, Internationalization, and University Rankings: Evaluating and Predicting Times Higher Education (THE) Data for Japan," Sustainability, MDPI, vol. 11(5), pages 1-12, March.
    11. Buckley, Ralf, 2019. "Tourism publications as newly tradeable commodities: Academic performance, prestige, power, competition, constraints and consents," Annals of Tourism Research, Elsevier, vol. 74(C), pages 121-133.
    12. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    13. Annamaria Demarinis Loiotile & Francesco De Nicolò & Adriana Agrimi & Loredana Bellantuono & Marianna La Rocca & Alfonso Monaco & Ester Pantaleo & Sabina Tangaro & Nicola Amoroso & Roberto Bellotti, 2022. "Best Practices in Knowledge Transfer: Insights from Top Universities," Sustainability, MDPI, vol. 14(22), pages 1-23, November.
    14. Enis Siniksaran & M. Hakan Satman, 2020. "WURS: a simulation software for university rankings—software review," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 701-717, January.
    15. Li Hongbo & Joseph Muiruri Thige & Ssali Max William, 2021. "Demystifying the University of Nairobi’s Academic Quality through Third Party Ranking Assessment," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 5(7), pages 765-772, July.
    16. Mónica Benito & Pilar Gil & Rosario Romera, 2019. "Funding, is it key for standing out in the university rankings?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 771-792, November.
    17. Meng-Chen Zhang & Bo-Wei Zhu & Chao-Meng Huang & Gwo-Hshiung Tzeng, 2021. "Systematic Evaluation Model for Developing Sustainable World-Class Universities: An East Asian Perspective," Mathematics, MDPI, vol. 9(8), pages 1-20, April.
    18. Zsuzsanna Banász & Zsolt T. Kosztyán & Vivien V. Csányi & András Telcs, 2023. "University leagues alongside rankings," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(1), pages 721-736, February.
    19. Li Hongbo & Joseph Muiruri Thige & Ssali Max William, 2021. "Demystifying the University of Nairobi’s Academic Quality through Third Party Ranking Assessment," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 5(07), pages 765-772, July.
    20. Tommaso Agasisti & Guo-liang Yang & Yao-yao Song & Carolyn-Thi Thanh Dung Tran, 2021. "Evaluating the higher education productivity of Chinese and European “elite” universities using a meta-frontier approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5819-5853, July.
    21. Gössling, Stefan & Moyle, Brent D. & Weaver, David, 2021. "Academic entrepreneurship: A bibliometric engagement model," Annals of Tourism Research, Elsevier, vol. 90(C).
    22. Syed Haider Khalil & Syed Mohsin Ali Shah & Fahad Sultan & Muhammad Ibrahim Khan & Sher Nawaz, 2023. "Categories and Institutional Change: Contesting the Uncontested Space Through National Rankings," SAGE Open, , vol. 13(3), pages 21582440231, September.
    23. Csató, László & Tóth, Csaba, 2020. "University rankings from the revealed preferences of the applicants," European Journal of Operational Research, Elsevier, vol. 286(1), pages 309-320.
    24. Benito, M. & Gil, P. & Romera, R., 2020. "Evaluating the influence of country characteristics on the Higher Education System Rankings’ progress," Journal of Informetrics, Elsevier, vol. 14(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Buckley, Ralf, 2019. "Tourism publications as newly tradeable commodities: Academic performance, prestige, power, competition, constraints and consents," Annals of Tourism Research, Elsevier, vol. 74(C), pages 121-133.
    2. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    3. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    4. Renato A. Villano & Carolyn‐Dung T. T. Tran, 2021. "Survey on technical efficiency in higher education: A meta‐fractional regression analysis," Pacific Economic Review, Wiley Blackwell, vol. 26(1), pages 110-135, February.
    5. Van Puyenbroeck, Tom & Rogge, Nicky, 2017. "Geometric mean quantity index numbers with Benefit-of-the-Doubt weights," European Journal of Operational Research, Elsevier, vol. 256(3), pages 1004-1014.
    6. Laurens CHERCHYE & Willem MOESEN & Nicky ROGGE & Tom VAN PUYENBROECK, 2009. "Constructing a knowledge economy composite indicator with imprecise data," Working Papers of Department of Economics, Leuven ces09.15, KU Leuven, Faculty of Economics and Business (FEB), Department of Economics, Leuven.
    7. Giménez, Víctor & Thieme, Claudio & Prior, Diego & Tortosa-Ausina, Emili, 2022. "Evaluation and determinants of preschool effectiveness in Chile," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    8. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    9. Guironnet, Jean-Pascal & Peypoch, Nicolas, 2018. "The geographical efficiency of education and research: The ranking of U.S. universities," Socio-Economic Planning Sciences, Elsevier, vol. 62(C), pages 44-55.
    10. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    11. Diogo Ferraz & Enzo B. Mariano & Daisy Rebelatto & Dominik Hartmann, 2020. "Linking Human Development and the Financial Responsibility of Regions: Combined Index Proposals Using Methods from Data Envelopment Analysis," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 150(2), pages 439-478, July.
    12. Maričić Milica & Bulajić Milica & Radojičić Zoran & Jeremić Veljko, 2016. "Multivariate approach to imposing additional constraints on the Benefit-of-the-Doubt model: The case of QS World University Rankings by Subject," Croatian Review of Economic, Business and Social Statistics, Sciendo, vol. 2(1), pages 1-14, September.
    13. Tom Puyenbroeck, 2018. "On the Output Orientation of the Benefit-of-the-Doubt-Model," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 139(2), pages 415-431, September.
    14. Ana Cárcaba & Eduardo González & Juan Ventura & Rubén Arrondo, 2017. "How Does Good Governance Relate to Quality of Life?," Sustainability, MDPI, vol. 9(4), pages 1-16, April.
    15. Tzeremes, Nickolaos & Halkos, George, 2010. "A DEA approach for measuring university departments’ efficiency," MPRA Paper 24029, University Library of Munich, Germany.
    16. Ana Cárcaba & Eduardo González & Juan Ventura, 2017. "Social Progress in Spanish Municipalities (2001–2011)," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 12(4), pages 997-1019, December.
    17. Hatefi, S.M. & Torabi, S.A., 2010. "A common weight MCDA-DEA approach to construct composite indicators," Ecological Economics, Elsevier, vol. 70(1), pages 114-120, November.
    18. Omrani, Hashem & Valipour, Mahsa & Jafari Mamakani, Saeid, 2019. "Construct a composite indicator based on integrating Common Weight Data Envelopment Analysis and principal component analysis models: An application for finding development degree of provinces in Iran," Socio-Economic Planning Sciences, Elsevier, vol. 68(C).
    19. Sunil Kumar Jauhar & Millie Pant & Richa Dutt, 2018. "Performance measurement of an Indian higher education institute: a sustainable educational supply chain management perspective," International Journal of System Assurance Engineering and Management, Springer;The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden, vol. 9(1), pages 180-193, February.
    20. Eduardo González & Ana Cárcaba & Juan Ventura, 2018. "Weight Constrained DEA Measurement of the Quality of Life in Spanish Municipalities in 2011," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 1157-1182, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:115:y:2018:i:1:d:10.1007_s11192-018-2666-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.