IDEAS home Printed from https://ideas.repec.org/a/eee/matcom/v154y2018icp48-64.html
   My bibliography  Save this article

Multiple-criteria performance ranking based on profile distributions: An application to university research evaluations

Author

Listed:
  • Kunsch, Pierre L.
  • Ishizaka, Alessio

Abstract

This article addresses a category of multi-criteria ranking problems in which performance evaluations of the ‘objects’ or ‘alternatives’ to be ranked are not given by unique numbers but by performance profile distributions running over an ordered set of performance score levels. The numerical values to be given to score levels are not specified a priori using cardinal scales. A weighted sum approach is developed based on order statistics to combine the individual profile distributions. In this way, a global ranking indicator is obtained, considering not only mean distribution values but also standard deviations. As a test of feasibility, the resulting Profile Ranking with Order Statistics Evaluations (PROSE) approach has been applied to the performance profile distributions provided by the UK Research Excellence Framework 2014, evaluating the quality of UK research.

Suggested Citation

  • Kunsch, Pierre L. & Ishizaka, Alessio, 2018. "Multiple-criteria performance ranking based on profile distributions: An application to university research evaluations," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 154(C), pages 48-64.
    • Handle: RePEc:eee:matcom:v:154:y:2018:i:c:p:48-64
    DOI: 10.1016/j.matcom.2018.05.021
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378475418301459
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.matcom.2018.05.021?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Corrente, Salvatore & Greco, Salvatore & Ishizaka, Alessio, 2016. "Combining analytical hierarchy process and Choquet integral within non-additive robust ordinal regression," Omega, Elsevier, vol. 61(C), pages 2-18.
    2. L-J Kao & C-C Lu & C-C Chiu, 2011. "The training institution efficiency of the semiconductor institute programme in Taiwan—application of spatiotemporal ICA with DEA approach," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(12), pages 2162-2172, December.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fabio Pugini, 2008. "The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(2), pages 225-244, August.
    4. Edwards, Ward & Barron, F. Hutton, 1994. "SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement," Organizational Behavior and Human Decision Processes, Elsevier, vol. 60(3), pages 306-325, December.
    5. Johnes, Jill, 2006. "Data envelopment analysis and its application to the measurement of efficiency in higher education," Economics of Education Review, Elsevier, vol. 25(3), pages 273-288, June.
    6. Roger Carrington & Tim Coelli & D. S. Prasada Rao, 2005. "The Performance Of Australian Universities: Conceptual Issues And Preliminary Results," Economic Papers, The Economic Society of Australia, vol. 24(2), pages 145-163, June.
    7. Pilar Murias & José Miguel & David Rodríguez, 2008. "A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(1), pages 129-146, October.
    8. Gerhard Kempkes & Carsten Pohl, 2010. "The efficiency of German universities-some evidence from nonparametric and parametric methods," Applied Economics, Taylor & Francis Journals, vol. 42(16), pages 2063-2079.
    9. L-J Kao & C-C Lu & C-C Chiu, 2011. "The training institution efficiency of the semiconductor institute programme in Taiwan—application of spatiotemporal ICA with DEA approach," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(12), pages 2162-2172, December.
    10. Lahdelma, Risto & Hokkanen, Joonas & Salminen, Pekka, 1998. "SMAA - Stochastic multiobjective acceptability analysis," European Journal of Operational Research, Elsevier, vol. 106(1), pages 137-143, April.
    11. Geraint Johnes, 2013. "Efficiency in English higher education institutions revisited: a network approach," Economics Bulletin, AccessEcon, vol. 33(4), pages 2698-2706.
    12. E Bayraktar & E Tatoglu & S Zaim, 2013. "Measuring the relative efficiency of quality management practices in Turkish public and private universities," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 64(12), pages 1810-1830, December.
    13. Marie-Laure Bougnol & José Dulá, 2006. "Validating DEA as a ranking tool: An application of DEA to assess performance in higher education," Annals of Operations Research, Springer, vol. 145(1), pages 339-365, July.
    14. Lahdelma, Risto & Miettinen, Kaisa & Salminen, Pekka, 2003. "Ordinal criteria in stochastic multicriteria acceptability analysis (SMAA)," European Journal of Operational Research, Elsevier, vol. 147(1), pages 117-127, May.
    15. Mustafa, A. & Goh, M., 1996. "Multi-criterion models for higher education administration," Omega, Elsevier, vol. 24(2), pages 167-178, April.
    16. Thomas R. Sexton & Christie L. Comunale & Stephen C. Gara, 2012. "Efficiency-Based Funding for Public Four-Year Colleges and Universities," Education Finance and Policy, MIT Press, vol. 7(3), pages 331-359, July.
    17. Politis, Y. & Siskos, Y., 2004. "Multicriteria methodology for the evaluation of a Greek engineering department," European Journal of Operational Research, Elsevier, vol. 156(1), pages 223-240, July.
    18. Johnes, Jill & Johnes, Geraint, 1995. "Research funding and performance in U.K. University Departments of Economics: A frontier analysis," Economics of Education Review, Elsevier, vol. 14(3), pages 301-314, September.
    19. Saaty, Thomas L. & Takizawa, Masahiro, 1986. "Dependence and independence: From linear hierarchies to nonlinear networks," European Journal of Operational Research, Elsevier, vol. 26(2), pages 229-237, August.
    20. C S Sarrico & R G Dyson, 2000. "Using DEA for planning in UK universities—an institutional perspective," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 51(7), pages 789-800, July.
    21. Tommaso Agasisti & Giuseppe Catalano & Paolo Landoni & Roberto Verganti, 2012. "Evaluating the performance of academic departments: an analysis of research-related output efficiency," Research Evaluation, Oxford University Press, vol. 21(1), pages 2-14, February.
    22. Johnes, Jill, 2006. "Measuring teaching efficiency in higher education: An application of data envelopment analysis to economics graduates from UK Universities 1993," European Journal of Operational Research, Elsevier, vol. 174(1), pages 443-456, October.
    23. Bana e Costa, Carlos A. & Oliveira, Mónica D., 2012. "A multicriteria decision analysis model for faculty evaluation," Omega, Elsevier, vol. 40(4), pages 424-436.
    24. E Thanassoulis & M Kortelainen & G Johnes & J Johnes, 2011. "Costs and efficiency of higher education institutions in England: a DEA analysis," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(7), pages 1282-1297, July.
    25. Rezaei, Jafar, 2015. "Best-worst multi-criteria decision-making method," Omega, Elsevier, vol. 53(C), pages 49-57.
    26. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "A field-standardized application of DEA to national-scale research assessment of universities," Journal of Informetrics, Elsevier, vol. 5(4), pages 618-628.
    27. Fandel, Gunter, 2007. "On the performance of universities in North Rhine-Westphalia, Germany: Government's redistribution of funds judged using DEA efficiency measures," European Journal of Operational Research, Elsevier, vol. 176(1), pages 521-533, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alessio Ishizaka & Enrique Mu, 2023. "What is so special about the analytic hierarchy and network process?," Annals of Operations Research, Springer, vol. 326(2), pages 625-634, July.
    2. Kunsch, Pierre L. & Ishizaka, Alessio, 2019. "A note on using centroid weights in additive multi-criteria decision analysis," European Journal of Operational Research, Elsevier, vol. 277(1), pages 391-393.
    3. Csató, László & Tóth, Csaba, 2020. "University rankings from the revealed preferences of the applicants," European Journal of Operational Research, Elsevier, vol. 286(1), pages 309-320.
    4. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.
    5. Pierre L. Kunsch & Jean-Pierre Brans, 2019. "Visualising multi-criteria weight elicitation by multiple stakeholders in complex decision systems," Operational Research, Springer, vol. 19(4), pages 955-971, December.
    6. Muñuzuri, Jesús & Muñoz-Díaz, María-Luisa, 2019. "Use of DEA to identify urban geographical zones with special difficulty for freight deliveries," Journal of Transport Geography, Elsevier, vol. 79(C), pages 1-1.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    2. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    3. Mehdi Rhaiem, 2017. "Measurement and determinants of academic research efficiency: a systematic review of the evidence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 581-615, February.
    4. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    5. Renato A. Villano & Carolyn‐Dung T. T. Tran, 2021. "Survey on technical efficiency in higher education: A meta‐fractional regression analysis," Pacific Economic Review, Wiley Blackwell, vol. 26(1), pages 110-135, February.
    6. Joanna Wolszczak-Derlacz & Aleksandra Parteka, 2011. "Efficiency of European public higher education institutions: a two-stage multicountry approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 887-917, December.
    7. Mad Ithnin Salleh & Nurul Fadly Habidin & Abdul Halim Masnan & Nordin Mamat, 2017. "Estimating Technical Efficiency and Bootstrapping Malmquist Indices: Analysis of Malaysian Preschool Sector," International Journal of Academic Research in Business and Social Sciences, Human Resource Management Academic Research Society, International Journal of Academic Research in Business and Social Sciences, vol. 7(3), pages 440-457, March.
    8. Agasisti, Tommaso & Barra, Cristian & Zotti, Roberto, 2016. "Evaluating the efficiency of Italian public universities (2008–2011) in presence of (unobserved) heterogeneity," Socio-Economic Planning Sciences, Elsevier, vol. 55(C), pages 47-58.
    9. Tzeremes, Nickolaos & Halkos, George, 2010. "A DEA approach for measuring university departments’ efficiency," MPRA Paper 24029, University Library of Munich, Germany.
    10. Tommaso Agasisti & Cristian Barra & Roberto Zotti, 2019. "Research, knowledge transfer, and innovation: The effect of Italian universities’ efficiency on local economic development 2006−2012," Journal of Regional Science, Wiley Blackwell, vol. 59(5), pages 819-849, November.
    11. Sanjeet Singh & Prabhat Ranjan, 2018. "Efficiency analysis of non-homogeneous parallel sub-unit systems for the performance measurement of higher education," Annals of Operations Research, Springer, vol. 269(1), pages 641-666, October.
    12. Amara, Nabil & Rhaiem, Mehdi & Halilem, Norrin, 2020. "Assessing the research efficiency of Canadian scholars in the management field: Evidence from the DEA and fsQCA," Journal of Business Research, Elsevier, vol. 115(C), pages 296-306.
    13. Sabine Gralka & Klaus Wohlrabe & Lutz Bornmann, 2018. "How to Measure Research Efficiency in Higher Education? Research Grants vs. Publication Output," CESifo Working Paper Series 7055, CESifo.
    14. Angel García Correas & Manuel Larran Jorge, 2010. "Análisis de diferentes medidas de la eficiencia investigadora y factores explicativos en las universidades públicas españolas," Investigaciones de Economía de la Educación volume 5, in: María Jesús Mancebón-Torrubia & Domingo P. Ximénez-de-Embún & José María Gómez-Sancho & Gregorio Gim (ed.), Investigaciones de Economía de la Educación 5, edition 1, volume 5, chapter 36, pages 703-732, Asociación de Economía de la Educación.
    15. Manuel Salas-Velasco, 2020. "Measuring and explaining the production efficiency of Spanish universities using a non-parametric approach and a bootstrapped-truncated regression," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 825-846, February.
    16. Bana e Costa, Carlos A. & Oliveira, Mónica D., 2012. "A multicriteria decision analysis model for faculty evaluation," Omega, Elsevier, vol. 40(4), pages 424-436.
    17. Barra, Cristian & Lagravinese, Raffaele & Zotti, Roberto, 2018. "Does econometric methodology matter to rank universities? An analysis of Italian higher education system," Socio-Economic Planning Sciences, Elsevier, vol. 62(C), pages 104-120.
    18. Yang, Guo-liang & Fukuyama, Hirofumi & Song, Yao-yao, 2018. "Measuring the inefficiency of Chinese research universities based on a two-stage network DEA model," Journal of Informetrics, Elsevier, vol. 12(1), pages 10-30.
    19. Marcel Clermont, 2016. "Effectiveness and efficiency of research in Germany over time: an analysis of German business schools between 2001 and 2009," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1347-1381, September.
    20. Erik E. Lehmann & Michele Meoli & Stefano Paleari & Sarah A. E. Stockinger, 2018. "Approaching effects of the economic crisis on university efficiency: a comparative study of Germany and Italy," Eurasian Business Review, Springer;Eurasia Business and Economics Society, vol. 8(1), pages 37-54, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:matcom:v:154:y:2018:i:c:p:48-64. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/mathematics-and-computers-in-simulation/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.