IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v12y2018i4p1235-1250.html
   My bibliography  Save this article

Evaluating university performance using reference point based composite indicators

Author

Listed:
  • El Gibari, Samira
  • Gómez, Trinidad
  • Ruiz, Francisco

Abstract

In this paper, we propose the application of a novel methodology to build composite indicators, in order to evaluate university performance. We analyse separately the three basic dimensions of our university system (research, teaching and technology transfer), because we are interested in getting a more accurate vision of each of them. In order to build the composite indicators, we use a multi-criteria analysis technique, based on the double reference point method. One advantage of this technique is the possibility to use reference levels, in such a way that the results obtained are easily interpreted in terms of the performance of the university with respect to these levels. Besides, aggregations for different compensation degrees are provided. In order to illustrate the advantages of this method, it has been applied to evaluate the performance of the public universities of the Spanish region of Andalucía, for year 2008. The results show that the performance of the Andalusian public universities in the teaching block is better than in the research and technology transfer blocks. The application lets us conclude that the methodology offers a warning system to assist in strategic decision making, and the values of the indicators allow us to find fields of improvement in all areas.

Suggested Citation

  • El Gibari, Samira & Gómez, Trinidad & Ruiz, Francisco, 2018. "Evaluating university performance using reference point based composite indicators," Journal of Informetrics, Elsevier, vol. 12(4), pages 1235-1250.
    • Handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1235-1250
    DOI: 10.1016/j.joi.2018.10.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157717303450
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2018.10.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Berbegal-Mirabent, Jasmina & Sánchez García, José Luís & Ribeiro-Soriano, D. Enrique, 2015. "University–industry partnerships for the provision of R&D services," Journal of Business Research, Elsevier, vol. 68(7), pages 1407-1413.
    2. Cabello, J.M. & Ruiz, F. & Pérez-Gladish, B. & Méndez-Rodríguez, P., 2014. "Synthetic indicators of mutual funds’ environmental responsibility: An application of the Reference Point Method," European Journal of Operational Research, Elsevier, vol. 236(1), pages 313-325.
    3. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2010. "Should you believe in the Shanghai ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 237-263, July.
    4. Daniel Torres-Salinas & Jose G. Moreno-Torres & Emilio Delgado-López-Cózar & Francisco Herrera, 2011. "A methodology for Institution-Field ranking based on a bidimensional analysis: the IFQ 2 A index," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 771-786, September.
    5. F Ruiz & J M Cabello & M Luque, 2011. "An application of reference point techniques to the calculation of synthetic sustainability indicators," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(1), pages 189-197, January.
    6. Peter W. A. West, 2009. "A Faustian bargain? Institutional responses to national and international rankings," Higher Education Management and Policy, OECD Publishing, vol. 21(1), pages 1-10.
    7. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2010. "Should you believe in the Shanghai ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 237-263, July.
    8. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Evaluating university research: Same performance indicator, different rankings," Journal of Informetrics, Elsevier, vol. 9(3), pages 514-525.
    9. repec:dau:papers:123456789/2947 is not listed on IDEAS
    10. Bonaccorsi, Andrea & Cicero, Tindaro, 2016. "Nondeterministic ranking of university departments," Journal of Informetrics, Elsevier, vol. 10(1), pages 224-237.
    11. Isidro F. Aguillo & Judit Bar-Ilan & Mark Levene & José Luis Ortega, 2010. "Comparing university rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 243-256, October.
    12. Pilar Murias & José Miguel & David Rodríguez, 2008. "A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(1), pages 129-146, October.
    13. Frenken, Koen & Heimeriks, Gaston J. & Hoekman, Jarno, 2017. "What drives university research performance? An analysis using the CWTS Leiden Ranking data," Journal of Informetrics, Elsevier, vol. 11(3), pages 859-872.
    14. De Filippo, D. & Casani, F. & Sanz-Casado, E., 2016. "University excellence initiatives in Spain, a possible strategy for optimising resources and improving local performance," Technological Forecasting and Social Change, Elsevier, vol. 113(PB), pages 185-194.
    15. Caldera, Aida & Debande, Olivier, 2010. "Performance of Spanish universities in technology transfer: An empirical analysis," Research Policy, Elsevier, vol. 39(9), pages 1160-1173, November.
    16. Fernández-Esquinas, Manuel & Pinto, Hugo & Yruela, Manuel Pérez & Pereira, Tiago Santos, 2016. "Tracing the flows of knowledge transfer: Latent dimensions and determinants of university–industry interactions in peripheral innovation systems," Technological Forecasting and Social Change, Elsevier, vol. 113(PB), pages 266-279.
    17. Oğuzhan Alaşehir & Murat Perit Çakır & Cengiz Acartürk & Nazife Baykal & Ural Akbulut, 2014. "URAP-TR: a national ranking for Turkish universities based on academic performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 159-178, October.
    18. BERBEGAL MIRABENT, Jasmina & SOLÉ PARELLADA, Francesc, 2012. "What Are We Measuring When Evaluating Universities’ Efficiency?," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. M. M. Segovia-González & I. Contreras, 2023. "A Composite Indicator to Compare the Performance of Male and Female Students in Educational Systems," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 165(1), pages 181-212, January.
    2. Samira El Gibari & Trinidad Gómez & Francisco Ruiz, 2022. "Combining reference point based composite indicators with data envelopment analysis: application to the assessment of universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4363-4395, August.
    3. Ana Garcia-Bernabeu & José Manuel Cabello & Francisco Ruiz, 2022. "A Reference Point-Based Proposal to Build Regional Quality of Life Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(1), pages 11-30, November.
    4. Ruiz, Francisco & El Gibari, Samira & Cabello, José M. & Gómez, Trinidad, 2020. "MRP-WSCI: Multiple reference point based weak and strong composite indicators," Omega, Elsevier, vol. 95(C).
    5. Muñoz, María M. & Rojas-de-Gracia, María-Mercedes & Navas-Sarasola, Carlos, 2022. "Measuring engagement on twitter using a composite index: An application to social media influencers," Journal of Informetrics, Elsevier, vol. 16(4).
    6. José Manuel Cabello & Francisco Ruiz & Blanca Pérez-Gladish, 2021. "An Alternative Aggregation Process for Composite Indexes: An Application to the Heritage Foundation Economic Freedom Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 153(2), pages 443-467, January.
    7. José Antonio, Palomero-González & Vicent, Almenar-Llongo & Ramón, Fuentes-Pascual, 2022. "A composite indicator index as a proxy for measuring the quality of water supply as perceived by users for urban water services," Technological Forecasting and Social Change, Elsevier, vol. 174(C).
    8. Ana Garcia-Bernabeu & José Manuel Cabello & Francisco Ruiz, 2020. "A Multi-Criteria Reference Point Based Approach for Assessing Regional Innovation Performance in Spain," Mathematics, MDPI, vol. 8(5), pages 1-21, May.
    9. Samira El Gibari & Carmen Perez-Esparrells & Trinidad Gomez & Francisco Ruiz, 2021. "Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach," Mathematics, MDPI, vol. 9(14), pages 1-34, July.
    10. Yelin Fu & Kong Xiangtianrui & Hao Luo & Lean Yu, 2020. "Constructing Composite Indicators with Collective Choice and Interval-Valued TOPSIS: The Case of Value Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 152(1), pages 117-135, November.
    11. Moehrle, Martin G. & Frischkorn, Jonas, 2021. "Bridge strongly or focus – An analysis of bridging patents in four application fields of carbon fiber reinforcements," Journal of Informetrics, Elsevier, vol. 15(2).
    12. Benito, M. & Gil, P. & Romera, R., 2020. "Evaluating the influence of country characteristics on the Higher Education System Rankings’ progress," Journal of Informetrics, Elsevier, vol. 14(3).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & Daniel Torres-Salinas & Francisco Herrera, 2012. "Ranking of research output of universities on the basis of the multidimensional prestige of influential fields: Spanish universities as a case of study," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 1081-1099, December.
    2. Elio Atenógenes Villaseñor & Ricardo Arencibia-Jorge & Humberto Carrillo-Calvet, 2017. "Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher education institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 77-104, January.
    3. Milica Jovanovic & Veljko Jeremic & Gordana Savic & Milica Bulajic & Milan Martic, 2012. "How does the normalization of data affect the ARWU ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(2), pages 319-327, November.
    4. Samira El Gibari & Carmen Perez-Esparrells & Trinidad Gomez & Francisco Ruiz, 2021. "Analyzing the Impact of Spanish University Funding Policies on the Evolution of Their Performance: A Multi-Criteria Approach," Mathematics, MDPI, vol. 9(14), pages 1-34, July.
    5. Samira El Gibari & Trinidad Gómez & Francisco Ruiz, 2022. "Combining reference point based composite indicators with data envelopment analysis: application to the assessment of universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4363-4395, August.
    6. Samira El Gibari & Trinidad Gómez & Francisco Ruiz, 2019. "Building composite indicators using multicriteria methods: a review," Journal of Business Economics, Springer, vol. 89(1), pages 1-24, February.
    7. Hiran H. Lathabai & Abhirup Nandy & Vivek Kumar Singh, 2021. "x-index: Identifying core competency and thematic research strengths of institutions using an NLP and network based ranking framework," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9557-9583, December.
    8. Enar Ruiz-Conde & Aurora Calderón-Martínez, 2014. "University institutional repositories: competitive environment and their role as communication media of scientific knowledge," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1283-1299, February.
    9. Veljko Jeremic & Milica Bulajic & Milan Martic & Zoran Radojicic, 2011. "A fresh approach to evaluating the academic ranking of world universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 587-596, June.
    10. Bana e Costa, Carlos A. & Oliveira, Mónica D., 2012. "A multicriteria decision analysis model for faculty evaluation," Omega, Elsevier, vol. 40(4), pages 424-436.
    11. Massucci, Francesco Alessandro & Docampo, Domingo, 2019. "Measuring the academic reputation through citation networks via PageRank," Journal of Informetrics, Elsevier, vol. 13(1), pages 185-201.
    12. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    13. Calabrese, Armando & Capece, Guendalina & Costa, Roberta & Di Pillo, Francesca & Giuffrida, Stefania, 2018. "A ‘power law’ based method to reduce size-related bias in indicators of knowledge performance: An application to university research assessment," Journal of Informetrics, Elsevier, vol. 12(4), pages 1263-1281.
    14. Daniel Torres-Salinas & Jose G. Moreno-Torres & Emilio Delgado-López-Cózar & Francisco Herrera, 2011. "A methodology for Institution-Field ranking based on a bidimensional analysis: the IFQ 2 A index," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 771-786, September.
    15. Antonio Fernández-Cano & Elvira Curiel-Marin & Manuel Torralbo-Rodríguez & Mónica Vallejo-Ruiz, 2018. "Questioning the Shanghai Ranking methodology as a tool for the evaluation of universities: an integrative review," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2069-2083, September.
    16. Sung-Shun Weng & Yang Liu & Yen-Ching Chuang, 2019. "Reform of Chinese Universities in the Context of Sustainable Development: Teacher Evaluation and Improvement Based on Hybrid Multiple Criteria Decision-Making Model," Sustainability, MDPI, vol. 11(19), pages 1-23, October.
    17. Alexandr Gedranovich & Mykhaylo Salnykov, 2012. "Productivity analysis of Belarusian higher education system," BEROC Working Paper Series 16, Belarusian Economic Research and Outreach Center (BEROC).
    18. Khatab Alqararah, 2023. "Assessing the robustness of composite indicators: the case of the Global Innovation Index," Journal of Innovation and Entrepreneurship, Springer, vol. 12(1), pages 1-22, December.
    19. Franceschini, Fiorenzo & Maisano, Domenico, 2011. "Structured evaluation of the scientific output of academic research groups by recent h-based indicators," Journal of Informetrics, Elsevier, vol. 5(1), pages 64-74.
    20. Zofio, Jose Luis & Aparicio, Juan & Barbero, Javier & Zabala-Iturriagagoitia, Jon Mikel, 2023. "The influence of bottlenecks on innovation systems performance: Put the slowest climber first," Technological Forecasting and Social Change, Elsevier, vol. 193(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:12:y:2018:i:4:p:1235-1250. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.