IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v167y2004i3p449-474.html
   My bibliography  Save this article

Statistical analysis of performance indicators in UK higher education

Author

Listed:
  • David Draper
  • Mark Gittoes

Abstract

Summary. Attempts to measure the quality with which institutions such as hospitals and universities carry out their public mandates have gained in frequency and sophistication over the last decade. We examine methods for creating performance indicators in multilevel or hierarchical settings (e.g. students nested within universities) based on a dichotomous outcome variable (e.g. drop‐out from the higher education system). The profiling methods that we study involve the indirect measurement of quality, by comparing institutional outputs after adjusting for inputs, rather than directly attempting to measure the quality of the processes unfolding inside the institutions. In the context of an extended case‐study of the creation of performance indicators for universities in the UK higher education system, we demonstrate the large sample functional equivalence between a method based on indirect standardization and an approach based on fixed effects hierarchical modelling, offer simulation results on the performance of the standardization method in null and non‐null settings, examine the sensitivity of this method to the inadvertent omission of relevant adjustment variables, explore random‐effects reformulations and characterize settings in which they are preferable to fixed effects hierarchical modelling in this type of quality assessment and discuss extensions to longitudinal quality modelling and the overall pros and cons of institutional profiling. Our results are couched in the language of higher education but apply with equal force to other settings with dichotomous response variables, such as the examination of observed and expected rates of mortality (or other adverse outcomes) in investigations of the quality of health care or the study of retention rates in the workplace.

Suggested Citation

  • David Draper & Mark Gittoes, 2004. "Statistical analysis of performance indicators in UK higher education," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 167(3), pages 449-474, August.
    • Handle: RePEc:bla:jorssa:v:167:y:2004:i:3:p:449-474
    DOI: 10.1111/j.1467-985X.2004.apm12.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-985X.2004.apm12.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1467-985X.2004.apm12.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Nicholas J. Horton & Garrett M. Fitzmaurice, 2002. "Maximum likelihood estimation of bivariate logistic models for incomplete responses with indicators of ignorable and non‐ignorable missingness," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 51(3), pages 281-295, July.
    2. C S Sarrico & R G Dyson, 2000. "Using DEA for planning in UK universities—an institutional perspective," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 51(7), pages 789-800, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paul Hewson & Keming Yu, 2008. "Quantile regression for binary performance indicators," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 24(5), pages 401-418, September.
    2. Adele H. Marshall & Mariangela Zenga & Aglaia Kalamatianou, 2020. "Academic Students’ Progress Indicators and Gender Gaps Based on Survival Analysis and Data Mining Frameworks," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 151(3), pages 1097-1128, October.
    3. Isabella Sulis & Mariano Porcu, 2015. "Assessing Divergences in Mathematics and Reading Achievement in Italian Primary Schools: A Proposal of Adjusted Indicators of School Effectiveness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 122(2), pages 607-634, June.
    4. Kato, Maki, 2023. "Careers of faculty with foreign degrees: The attributes and impact on academic ranks in Japan," International Journal of Educational Development, Elsevier, vol. 99(C).
    5. Isabella Sulis & Mariano Porcu & Vincenza Capursi, 2019. "On the Use of Student Evaluation of Teaching: A Longitudinal Analysis Combining Measurement Issues and Implications of the Exercise," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 142(3), pages 1305-1331, April.
    6. F. Cugnata & G. Perucca & S. Salini, 2017. "Bayesian networks and the assessment of universities' value added," Journal of Applied Statistics, Taylor & Francis Journals, vol. 44(10), pages 1785-1806, July.
    7. Columbu, Silvia & Porcu, Mariano & Sulis, Isabella, 2021. "University choice and the attractiveness of the study area: Insights on the differences amongst degree programmes in Italy based on generalised mixed-effect models," Socio-Economic Planning Sciences, Elsevier, vol. 74(C).
    8. Sulis, Isabella & Giambona, Francesca & Porcu, Mariano, 2020. "Adjusted indicators of quality and equity for monitoring the education systems over time. Insights on EU15 countries from PISA surveys," Socio-Economic Planning Sciences, Elsevier, vol. 69(C).
    9. Maria Cristiana Martini & Luigi Fabbris, 2017. "Beyond Employment Rate: A Multidimensional Indicator of Higher Education Effectiveness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 130(1), pages 351-370, January.
    10. Sarrico, Cláudia S. & Teixeira, Pedro N. & Rosa, Maria J. & Cardoso, Margarida F., 2009. "Subject mix and productivity in Portuguese universities," European Journal of Operational Research, Elsevier, vol. 197(1), pages 287-295, August.
    11. Francesca Giambona & Mariano Porcu & Isabella Sulis, 2023. "Does education protect families' well-being in times of crisis? Measurement issues and empirical findings from IT-SILC data," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 32(1), pages 299-328, March.
    12. Silvia Bianconcini & Silvia Cagnone, 2012. "A General Multivariate Latent Growth Model With Applications to Student Achievement," Journal of Educational and Behavioral Statistics, , vol. 37(2), pages 339-364, April.
    13. Isabella Sulis & Mariano Porcu, 2012. "Comparing degree programs from students’ assessments: A LCRA-based adjusted composite indicator," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 21(2), pages 193-209, June.
    14. Tobias Wolbring, 2012. "Class Attendance and Students’ Evaluations of Teaching," Evaluation Review, , vol. 36(1), pages 72-96, February.
    15. David I. Ohlssen & Linda D. Sharples & David J. Spiegelhalter, 2007. "A hierarchical modelling framework for identifying unusual performance in health care providers," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 865-890, October.
    16. Nicholas Longford & D. B. Rubin, 2006. "Performance assessment and league tables. Comparing like with like," Economics Working Papers 994, Department of Economics and Business, Universitat Pompeu Fabra.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    2. George E. Halkos & Nickolaos G. Tzeremes & Stavros A. Kourtzidis, 2010. "An application of statistical interference in DEA models: An analysis of public owned university departments' efficiency," EERI Research Paper Series EERI_RP_2010_17, Economics and Econometrics Research Institute (EERI), Brussels.
    3. Xiang Ji & Jie Wu & Qingyuan Zhu & Jiasen Sun, 2019. "Using a hybrid heterogeneous DEA method to benchmark China’s sustainable urbanization: an empirical study," Annals of Operations Research, Springer, vol. 278(1), pages 281-335, July.
    4. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    5. Robert G. Dyson & Frances A. O’Brien & Devan B. Shah, 2021. "Soft OR and Practice: The Contribution of the Founders of Operations Research," Operations Research, INFORMS, vol. 69(3), pages 727-738, May.
    6. Sarrico, C. S. & Dyson, R. G., 2004. "Restricting virtual weights in data envelopment analysis," European Journal of Operational Research, Elsevier, vol. 159(1), pages 17-34, November.
    7. Johnes, Jill, 2015. "Operational Research in education," European Journal of Operational Research, Elsevier, vol. 243(3), pages 683-696.
    8. Tzeremes, Nickolaos & Halkos, George, 2010. "A DEA approach for measuring university departments’ efficiency," MPRA Paper 24029, University Library of Munich, Germany.
    9. Ahti Salo & Antti Punkka, 2011. "Ranking Intervals and Dominance Relations for Ratio-Based Efficiency Analysis," Management Science, INFORMS, vol. 57(1), pages 200-214, January.
    10. Fernando Gascón & Jesús Lozano & Borja Ponte & David Fuente, 2017. "Measuring the efficiency of large pharmaceutical companies: an industry analysis," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 18(5), pages 587-608, June.
    11. Colin Glass, J. & McCallion, Gillian & McKillop, Donal G. & Rasaratnam, Syamarlah & Stringer, Karl S., 2006. "Implications of variant efficiency measures for policy evaluations in UK higher education," Socio-Economic Planning Sciences, Elsevier, vol. 40(2), pages 119-142, June.
    12. Jill Johnes & Swati Virmani, 2020. "Chief executive pay in UK higher education: the role of university performance," Annals of Operations Research, Springer, vol. 288(2), pages 547-576, May.
    13. Yi-Chieh Chen & Lin-Huan Hu & Wan Chen Lu & Jei-Zheng Wu & Jiun-Jen Yang, 2021. "Multiple Criteria Decision-Making for Developing an International Game Participation Strategy: A Novel Application of the Data Envelopment Analysis (DEA) Two-Stage Efficiency Process," Mathematics, MDPI, vol. 9(14), pages 1-16, July.
    14. Sanjeet Singh & Prabhat Ranjan, 2018. "Efficiency analysis of non-homogeneous parallel sub-unit systems for the performance measurement of higher education," Annals of Operations Research, Springer, vol. 269(1), pages 641-666, October.
    15. Mad Ithnin Salleh & Nurul Fadly Habidin & Abdul Halim Masnan & Nordin Mamat, 2017. "Estimating Technical Efficiency and Bootstrapping Malmquist Indices: Analysis of Malaysian Preschool Sector," International Journal of Academic Research in Business and Social Sciences, Human Resource Management Academic Research Society, International Journal of Academic Research in Business and Social Sciences, vol. 7(3), pages 440-457, March.
    16. Lu, Wen-Min, 2012. "Intellectual capital and university performance in Taiwan," Economic Modelling, Elsevier, vol. 29(4), pages 1081-1089.
    17. Khoshnevis, Pegah & Teirlinck, Peter, 2018. "Performance evaluation of R&D active firms," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 16-28.
    18. Kunsch, Pierre L. & Ishizaka, Alessio, 2018. "Multiple-criteria performance ranking based on profile distributions: An application to university research evaluations," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 154(C), pages 48-64.
    19. Mehdi Rhaiem, 2017. "Measurement and determinants of academic research efficiency: a systematic review of the evidence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 581-615, February.
    20. R G Dyson & E A Shale, 2010. "Data envelopment analysis, operational research and uncertainty," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 61(1), pages 25-34, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:167:y:2004:i:3:p:449-474. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.