IDEAS home Printed from https://ideas.repec.org/a/pal/jorsoc/v68y2017i4d10.1057_s41274-016-0168-1.html
   My bibliography  Save this article

Assessing research effectiveness: a comparison of alternative nonparametric models

Author

Listed:
  • Giannis Karagiannis

    (University of Macedonia)

  • Georgia Paschalidou

    (University of Macedonia)

Abstract

In this paper, we examine three alternative a posteriori weighting schemes with variable, common and restricted weights in order to assess research productivity by means of two seemingly similar nonparametric models: the Benefit-of-the-doubt and the Kao and Hung (2003) model. Our empirical results, based on different types of faculty members’ publications, show that there is more variability in the estimated effectiveness scores among alternative weighting schemes within each model rather than between models for any particular weighting scheme. In addition, we also found that the effectiveness scores from the BoD model are greater than or equal to those from the K&H model for the variable- and the restricted-weights schemes while there is no clear pattern between the BoD and the K&H effectiveness scores from the common-weights scheme.

Suggested Citation

  • Giannis Karagiannis & Georgia Paschalidou, 2017. "Assessing research effectiveness: a comparison of alternative nonparametric models," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 456-468, April.
    • Handle: RePEc:pal:jorsoc:v:68:y:2017:i:4:d:10.1057_s41274-016-0168-1
    DOI: 10.1057/s41274-016-0168-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41274-016-0168-1
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1057/s41274-016-0168-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lovell, C. A. Knox & Pastor, Jesus T., 1999. "Radial DEA models without inputs or without outputs," European Journal of Operational Research, Elsevier, vol. 118(1), pages 46-51, October.
    2. Giannis Karagiannis, 2017. "On aggregate composite indicators," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(7), pages 741-746, July.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Tindaro Cicero, 2012. "What is the appropriate length of the publication period over which to assess research performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 1005-1017, December.
    4. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    5. Lins, Marcos P. Estellita & Gomes, Eliane G. & Soares de Mello, Joao Carlos C. B. & Soares de Mello, Adelino Jose R., 2003. "Olympic ranking based on a zero sum gains DEA model," European Journal of Operational Research, Elsevier, vol. 148(2), pages 312-322, July.
    6. Wade D. Cook & Joe Zhu, 2015. "DEA Cross Efficiency," International Series in Operations Research & Management Science, in: Joe Zhu (ed.), Data Envelopment Analysis, edition 127, chapter 2, pages 23-43, Springer.
    7. Karagiannis, Giannis & Knox Lovell, C.A., 2016. "Productivity measurement in radial DEA models with a single constant input," European Journal of Operational Research, Elsevier, vol. 251(1), pages 323-328.
    8. Färe, Rolf & Karagiannis, Giannis, 2017. "The denominator rule for share-weighting aggregation," European Journal of Operational Research, Elsevier, vol. 260(3), pages 1175-1180.
    9. Pilar Murias & José Miguel & David Rodríguez, 2008. "A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(1), pages 129-146, October.
    10. Doyle, John R., 1995. "Multiattribute Choice for the Lazy Decision Maker: Let the Alternatives Decide!," Organizational Behavior and Human Decision Processes, Elsevier, vol. 62(1), pages 87-100, April.
    11. Liu, W.B. & Zhang, D.Q. & Meng, W. & Li, X.X. & Xu, F., 2011. "A study of DEA models without explicit inputs," Omega, Elsevier, vol. 39(5), pages 472-480, October.
    12. Caporaletti, L. E. & Dulá, J. H. & Womer, N. K., 1999. "Performance evaluation based on multiple attributes with nonparametric frontiers," Omega, Elsevier, vol. 27(6), pages 637-645, December.
    13. Kao, Chiang & Wu, Wann-Yih & Hsieh, Wen-Jen & Wang, Tai-Yue & Lin, Chinho & Chen, Liang-Hsuan, 2008. "Measuring the national competitiveness of Southeast Asian countries," European Journal of Operational Research, Elsevier, vol. 187(2), pages 613-628, June.
    14. Kao, Chiang, 1994. "Evaluation of junior colleges of technology: The Taiwan case," European Journal of Operational Research, Elsevier, vol. 72(1), pages 43-51, January.
    15. Sungmook Lim & Joe Zhu, 2015. "DEA cross-efficiency evaluation under variable returns to scale," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 66(3), pages 476-487, March.
    16. Kao, Chiang & Hung, Hsi-Tai, 2007. "Management performance: An empirical study of the manufacturing companies in Taiwan," Omega, Elsevier, vol. 35(2), pages 152-160, April.
    17. Laurens Cherchye & Willem Moesen & Nicky Rogge & Tom Puyenbroeck, 2007. "An Introduction to ‘Benefit of the Doubt’ Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 82(1), pages 111-145, May.
    18. Liang, Liang & Wu, Jie & Cook, Wade D. & Zhu, Joe, 2008. "Alternative secondary goals in DEA cross-efficiency evaluation," International Journal of Production Economics, Elsevier, vol. 113(2), pages 1025-1030, June.
    19. Ng, Wan Lung, 2008. "An efficient and simple model for multiple criteria supplier selection problem," European Journal of Operational Research, Elsevier, vol. 186(3), pages 1059-1067, May.
    20. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "A field-standardized application of DEA to national-scale research assessment of universities," Journal of Informetrics, Elsevier, vol. 5(4), pages 618-628.
    21. T Joro & E-J Viitala, 2004. "Weight-restricted DEA in action: from expert opinions to mathematical models," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 55(8), pages 814-821, August.
    22. Hagen, Nils T., 2014. "Counting and comparing publication output with and without equalizing and inflationary bias," Journal of Informetrics, Elsevier, vol. 8(2), pages 310-317.
    23. Ng, Wan Lung, 2007. "A simple classifier for multiple criteria ABC analysis," European Journal of Operational Research, Elsevier, vol. 177(1), pages 344-353, February.
    24. Oral, Muhittin & Oukil, Amar & Malouin, Jean-Louis & Kettani, Ossama, 2014. "The appreciative democratic voice of DEA: A case of faculty academic performance evaluation," Socio-Economic Planning Sciences, Elsevier, vol. 48(1), pages 20-28.
    25. Sungmook Lim & Joe Zhu, 2015. "DEA Cross Efficiency Under Variable Returns to Scale," International Series in Operations Research & Management Science, in: Joe Zhu (ed.), Data Envelopment Analysis, edition 127, chapter 3, pages 45-66, Springer.
    26. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fabio Pugini, 2008. "The measurement of Italian universities’ research productivity by a non parametric-bibliometric methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(2), pages 225-244, August.
    27. Angel Prieto & José ZofIo, 2001. "Evaluating Effectiveness in Public Provision of Infrastructure and Equipment: The Case of Spanish Municipalities," Journal of Productivity Analysis, Springer, vol. 15(1), pages 41-58, January.
    28. Chiang Kao & Hwei-Lan Pao, 2009. "An evaluation of research performance in management of 168 Taiwan universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(2), pages 261-277, February.
    29. Chiang Kao & Shiang-Tai Liu & Hwei-Lan Pao, 2012. "Assessing improvement in management research in Taiwan," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(1), pages 75-87, July.
    30. Kristof Witte & Nicky Rogge, 2010. "To publish or not to publish? On the aggregation and drivers of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 657-680, December.
    31. Kristof Witte & Lenka Hudrlikova, 2013. "What about excellence in teaching? A benevolent ranking of universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 337-364, July.
    32. Wang, Ying-Ming & Chin, Kwai-Sang, 2010. "Some alternative models for DEA cross-efficiency evaluation," International Journal of Production Economics, Elsevier, vol. 128(1), pages 332-338, November.
    33. Emmanuel Thanassoulis & Kristof Witte & Jill Johnes & Geraint Johnes & Giannis Karagiannis & Conceição S. Portela, 2016. "Applications of Data Envelopment Analysis in Education," International Series in Operations Research & Management Science, in: Joe Zhu (ed.), Data Envelopment Analysis, chapter 0, pages 367-438, Springer.
    34. Soares de Mello, João Carlos C.B. & Angulo Meza, Lidia & da Silveira, Juliana Quintanilha & Gomes, Eliane Gonçalves, 2013. "About negative efficiencies in Cross Evaluation BCC input oriented models," European Journal of Operational Research, Elsevier, vol. 229(3), pages 732-737.
    35. C Kao & H-T Hung, 2005. "Data envelopment analysis with common weights: the compromise solution approach," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 56(10), pages 1196-1203, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thanassoulis, E. & Sotiros, D. & Koronakos, G. & Despotis, D., 2018. "Assessing the cost-effectiveness of university academic recruitment and promotion policies," European Journal of Operational Research, Elsevier, vol. 264(2), pages 742-755.
    2. Pereira, Miguel Alves & Camanho, Ana Santos & Figueira, José Rui & Marques, Rui Cunha, 2021. "Incorporating preference information in a range directional composite indicator: The case of Portuguese public hospitals," European Journal of Operational Research, Elsevier, vol. 294(2), pages 633-650.
    3. Pereira, Miguel Alves & Camanho, Ana Santos & Marques, Rui Cunha & Figueira, José Rui, 2021. "The convergence of the World Health Organization Member States regarding the United Nations’ Sustainable Development Goal ‘Good health and well-being’," Omega, Elsevier, vol. 104(C).
    4. Giuseppe Coco & Raffaele Lagravinese & Giuliano Resce, 2020. "Beyond the weights: a multicriteria approach to evaluate inequality in education," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 18(4), pages 469-489, December.
    5. Cristian Barra & Raffaele Lagravinese & Roberto Zotti, 2022. "Exploring hospital efficiency within and between Italian regions: new empirical evidence," Journal of Productivity Analysis, Springer, vol. 57(3), pages 269-284, June.
    6. Fu, Tsu-Tan & See, Kok Fong, 2022. "An integrated analysis of quality and productivity growth in China’s and Taiwan’s higher education institutions," Economic Analysis and Policy, Elsevier, vol. 74(C), pages 234-249.
    7. Giannis Karagiannis & Suzanna M. Paleologou, 2021. "A regression-based improvement to the multiple criteria ABC inventory classification analysis," Annals of Operations Research, Springer, vol. 306(1), pages 369-382, November.
    8. Rubo Zhao & Yixiang Tian & Ao Lei & Francis Boadu & Ze Ren, 2019. "The Effect of Local Government Debt on Regional Economic Growth in China: A Nonlinear Relationship Approach," Sustainability, MDPI, vol. 11(11), pages 1-22, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ebrahimi, Bohlool & Dhamotharan, Lalitha & Ghasemi, Mohammad Reza & Charles, Vincent, 2022. "A cross-inefficiency approach based on the deviation variables framework," Omega, Elsevier, vol. 111(C).
    2. Chiang Kao & Shiang-Tai Liu & Hwei-Lan Pao, 2012. "Assessing improvement in management research in Taiwan," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(1), pages 75-87, July.
    3. Ali Homayoni & Reza Fallahnejad & Farhad Hosseinzadeh Lotfi, 2022. "Cross Malmquist Productivity Index in Data Envelopment Analysis," 4OR, Springer, vol. 20(4), pages 567-602, December.
    4. Van Puyenbroeck, Tom & Rogge, Nicky, 2017. "Geometric mean quantity index numbers with Benefit-of-the-Doubt weights," European Journal of Operational Research, Elsevier, vol. 256(3), pages 1004-1014.
    5. Karagiannis, Roxani & Karagiannis, Giannis, 2018. "Intra- and inter-group composite indicators using the BoD model," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 44-51.
    6. Feng Li & Han Wu & Qingyuan Zhu & Liang Liang & Gang Kou, 2021. "Data envelopment analysis cross efficiency evaluation with reciprocal behaviors," Annals of Operations Research, Springer, vol. 302(1), pages 173-210, July.
    7. Kao, Chiang & Liu, Shiang-Tai, 2020. "A slacks-based measure model for calculating cross efficiency in data envelopment analysis," Omega, Elsevier, vol. 95(C).
    8. Giannis Karagiannis & Suzanna M. Paleologou, 2021. "A regression-based improvement to the multiple criteria ABC inventory classification analysis," Annals of Operations Research, Springer, vol. 306(1), pages 369-382, November.
    9. Färe, Rolf & Karagiannis, Giannis, 2014. "Benefit-of-the-doubt aggregation and the diet problem," Omega, Elsevier, vol. 47(C), pages 33-35.
    10. Vincent Charles & Guillermo Díaz, 2017. "A Non-radial DEA Index for Peruvian Regional Competitiveness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 134(2), pages 747-770, November.
    11. Ravanos, Panagiotis & Karagiannis, Giannis, 2022. "Tricks with the BoD model and an application to the e-Government Development Index," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    12. José L. Ruiz & Inmaculada Sirvent, 2017. "Fuzzy cross-efficiency evaluation: a possibility approach," Fuzzy Optimization and Decision Making, Springer, vol. 16(1), pages 111-126, March.
    13. Nguyen, Trang Thi Thuy & Pham, Binh Thai & Prior, Diego & van Hemmen, Stefan, 2022. "Performance of tax simplification around the world: A panel frontier analysis," Socio-Economic Planning Sciences, Elsevier, vol. 80(C).
    14. Rogge, Nicky, 2018. "Composite indicators as generalized benefit-of-the-doubt weighted averages," European Journal of Operational Research, Elsevier, vol. 267(1), pages 381-392.
    15. Balk, Bert M. & (René) De Koster, M.B.M. & Kaps, Christian & Zofío, José L., 2021. "An evaluation of cross-efficiency methods: With an application to warehouse performance," Applied Mathematics and Computation, Elsevier, vol. 406(C).
    16. Andreas Dellnitz & Elmar Reucher & Andreas Kleine, 2021. "Efficiency evaluation in data envelopment analysis using strong defining hyperplanes," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 43(2), pages 441-465, June.
    17. Oukil, Amar, 2020. "Exploiting value system multiplicity and preference voting for robust ranking," Omega, Elsevier, vol. 94(C).
    18. Jie Wu & Junfei Chu & Qingyuan Zhu & Yongjun Li & Liang Liang, 2016. "Determining common weights in data envelopment analysis based on the satisfaction degree," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 67(12), pages 1446-1458, December.
    19. Aparicio, Juan & Zofío, José L., 2021. "Economic cross-efficiency," Omega, Elsevier, vol. 100(C).
      • Aparicio, J. & Zofío, J.L., 2019. "Economic Cross-Efficiency," ERIM Report Series Research in Management ERS-2019-001-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    20. Ramón, Nuria & Ruiz, José L. & Sirvent, Inmaculada, 2020. "Cross-benchmarking for performance evaluation: Looking across best practices of different peer groups using DEA," Omega, Elsevier, vol. 92(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:jorsoc:v:68:y:2017:i:4:d:10.1057_s41274-016-0168-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.palgrave-journals.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.