IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v11y2017i3p613-628.html
   My bibliography  Save this article

Are there any frontiers of research performance? Efficiency measurement of funded research projects with the Bayesian stochastic frontier analysis for count data

Author

Listed:
  • Mutz, Rüdiger
  • Bornmann, Lutz
  • Daniel, Hans-Dieter

Abstract

In recent years, scientometrics has devoted increasing attention to the question of measurement of productivity and efficiency in research. In econometrics, the question is usually examined using data envelopment analysis. Alternatively, in this paper we propose using a statistical approach, Bayesian stochastic frontier analysis (B-SFA), that explicitly considers the stochastic nature of (count) data. The Austrian Science Fund (FWF) made data available to us from their peer review process (ex-ante peer evaluation of proposals, final research product reports) and bibliometric data. The data analysis was done for a subsample of N=1,046 FWF-funded projects (in Life Science and Medicine, Formal and Physical Sciences). For two outcome variables, a general latent research product dimension (CFACTOR) and the total number of publications (P), technical efficiency values (TE) were estimated for each project using the SFA production functions. The TE values for CFACTOR and P were on average 0.86 and 0.27, as compared with a maximum TE value of 1.0. With regard to CFACTOR, female PIs, younger PIs, and projects with longer durations have slightly higher TE than male PIs, older PIs, and projects with shorter durations. A simulation study showed the statistical behavior of the procedure under different sampling conditions.

Suggested Citation

  • Mutz, Rüdiger & Bornmann, Lutz & Daniel, Hans-Dieter, 2017. "Are there any frontiers of research performance? Efficiency measurement of funded research projects with the Bayesian stochastic frontier analysis for count data," Journal of Informetrics, Elsevier, vol. 11(3), pages 613-628.
    • Handle: RePEc:eee:infome:v:11:y:2017:i:3:p:613-628
    DOI: 10.1016/j.joi.2017.04.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157716303741
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2017.04.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Esmeralda Ramalho & Joaquim Ramalho & Pedro Henriques, 2010. "Fractional regression models for second stage DEA efficiency analyses," Journal of Productivity Analysis, Springer, vol. 34(3), pages 239-255, December.
    2. Chien-Ming Chen & Magali A. Delmas & Marvin B. Lieberman, 2015. "Production frontier methodologies and efficiency as a performance measure in strategic management research," Strategic Management Journal, Wiley Blackwell, vol. 36(1), pages 19-36, January.
    3. Antreas Athanassopoulos & Estelle Shale, 1997. "Assessing the Comparative Efficiency of Higher Education Institutions in the UK by the Means of Data Envelopment Analysis," Education Economics, Taylor & Francis Journals, vol. 5(2), pages 117-134.
    4. Vincent Larivière & Chaoqun Ni & Yves Gingras & Blaise Cronin & Cassidy R. Sugimoto, 2013. "Bibliometrics: Global gender disparities in science," Nature, Nature, vol. 504(7479), pages 211-213, December.
    5. Fried, Harold O. & Lovell, C. A. Knox & Schmidt, Shelton S. (ed.), 2008. "The Measurement of Productive Efficiency and Productivity Growth," OUP Catalogue, Oxford University Press, number 9780195183528.
    6. Efthymios G. Tsionas, 2002. "Stochastic frontier models with random coefficients," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 17(2), pages 127-147.
    7. Michael J. Stringer & Marta Sales-Pardo & Luís A. Nunes Amaral, 2010. "Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(7), pages 1377-1385, July.
    8. Liu, John S. & Lu, Louis Y.Y. & Lu, Wen-Min & Lin, Bruce J.Y., 2013. "A survey of DEA applications," Omega, Elsevier, vol. 41(5), pages 893-902.
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2014. "How do you define and measure research productivity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1129-1144, November.
    10. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    11. Hilbe,Joseph M., 2014. "Modeling Count Data," Cambridge Books, Cambridge University Press, number 9781107028333.
    12. Hung-jen Wang & Peter Schmidt, 2002. "One-Step and Two-Step Estimation of the Effects of Exogenous Variables on Technical Efficiency Levels," Journal of Productivity Analysis, Springer, vol. 18(2), pages 129-144, September.
    13. Drivas, Kyriakos & Economidou, Claire & Tsionas, Efthymios G., 2014. "A Poisson Stochastic Frontier Model with Finite Mixture Structure," MPRA Paper 57485, University Library of Munich, Germany.
    14. Eduardo Fé & Richard Hofler, 2013. "Count data stochastic frontier models, with an application to the patents–R&D relationship," Journal of Productivity Analysis, Springer, vol. 39(3), pages 271-284, June.
    15. Léopold Simar & Paul Wilson, 2011. "Two-stage DEA: caveat emptor," Journal of Productivity Analysis, Springer, vol. 36(2), pages 205-218, October.
    16. Robert B. Archibald & David H. Feldman, 2006. "Graduation Rates and Accountability: Regressions versus Production Frontiers," Working Papers 24, Department of Economics, College of William and Mary.
    17. Ricardo S. Ehlers, 2011. "Comparison of Bayesian models for production efficiency," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(11), pages 2433-2443, January.
    18. Maria Olivares & Heike Wetzel, 2014. "Editor's Choice Competing in the Higher Education Market: Empirical Evidence for Economies of Scale and Scope in German Higher Education Institutions," CESifo Economic Studies, CESifo, vol. 60(4), pages 653-680.
    19. Jim Griffin & Mark Steel, 2007. "Bayesian stochastic frontier analysis using WinBUGS," Journal of Productivity Analysis, Springer, vol. 27(3), pages 163-176, June.
    20. Giovanni Abramo & Ciriaco Andrea D’Angelo & Gianluca Murgia, 2016. "The combined effects of age and seniority on research performance of full professors," Science and Public Policy, Oxford University Press, vol. 43(3), pages 301-319.
    21. Johnson, Andrew L. & Kuosmanen, Timo, 2012. "One-stage and two-stage DEA estimation of the effects of contextual variables," European Journal of Operational Research, Elsevier, vol. 220(2), pages 559-570.
    22. van den Broeck, Julien & Koop, Gary & Osiewalski, Jacek & Steel, Mark F. J., 1994. "Stochastic frontier models : A Bayesian perspective," Journal of Econometrics, Elsevier, vol. 61(2), pages 273-303, April.
    23. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S. & van Raan, Anthony F.J., 2011. "Towards a new crown indicator: Some theoretical considerations," Journal of Informetrics, Elsevier, vol. 5(1), pages 37-47.
    24. Johnes, Jill, 2006. "Data envelopment analysis and its application to the measurement of efficiency in higher education," Economics of Education Review, Elsevier, vol. 25(3), pages 273-288, June.
    25. Liu, John S. & Lu, Louis Y.Y. & Lu, Wen-Min & Lin, Bruce J.Y., 2013. "Data envelopment analysis 1978–2010: A citation-based literature survey," Omega, Elsevier, vol. 41(1), pages 3-15.
    26. Geraint Johnes & Jill Johnes (ed.), 2004. "International Handbook on the Economics of Education," Books, Edward Elgar Publishing, number 2847.
    27. Kumbhakar,Subal C. & Wang,Hung-Jen & Horncastle,Alan P., 2015. "A Practitioner's Guide to Stochastic Frontier Analysis Using Stata," Cambridge Books, Cambridge University Press, number 9781107609464.
    28. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    29. Tammy Harris & Joseph M. Hilbe & James W. Hardin, 2014. "Modeling count data with generalized distributions," Stata Journal, StataCorp LP, vol. 14(3), pages 562-579, September.
    30. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2016. "A farewell to the MNCS and like size-independent indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 646-651.
    31. Jill Johnes, 2004. "Efficiency Measurement," Chapters, in: Geraint Johnes & Jill Johnes (ed.), International Handbook on the Economics of Education, chapter 16, Edward Elgar Publishing.
    32. Willam Greene, 2005. "Fixed and Random Effects in Stochastic Frontier Models," Journal of Productivity Analysis, Springer, vol. 23(1), pages 7-32, January.
    33. Kyriakos Drivas & Athanasios T. Balafoutis & Stelios Rozakis, 2014. "Research Funding and Academic Output: The Case of Agricultural University of Athens," Working Papers 2014-5, Agricultural University of Athens, Department Of Agricultural Economics.
    34. Hugo Horta & T Austin Lacy, 2011. "How does size matter for science? Exploring the effects of research unit size on academics' scientific productivity and information exchange behaviors," Science and Public Policy, Oxford University Press, vol. 38(6), pages 449-460, July.
    35. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "Towards a new crown indicator: an empirical analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 467-481, June.
    36. Abbott, M. & Doucouliagos, C., 2003. "The efficiency of Australian universities: a data envelopment analysis," Economics of Education Review, Elsevier, vol. 22(1), pages 89-97, February.
    37. Bornmann, Lutz & Stefaner, Moritz & de Moya Anegón, Felix & Mutz, Rüdiger, 2016. "Excellence networks in science: A Web-based application based on Bayesian multilevel logistic regression (BMLR) for the identification of institutions collaborating successfully," Journal of Informetrics, Elsevier, vol. 10(1), pages 312-327.
    38. Bolli, Thomas & Olivares, Maria & Bonaccorsi, Andrea & Daraio, Cinzia & Aracil, Adela Garcia & Lepori, Benedetto, 2016. "The differential effects of competitive funding on the production frontier and the efficiency of universities," Economics of Education Review, Elsevier, vol. 52(C), pages 91-104.
    39. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "A field-standardized application of DEA to national-scale research assessment of universities," Journal of Informetrics, Elsevier, vol. 5(4), pages 618-628.
    40. Parmeter, Christopher F. & Kumbhakar, Subal C., 2014. "Efficiency Analysis: A Primer on Recent Advances," Foundations and Trends(R) in Econometrics, now publishers, vol. 7(3-4), pages 191-385, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bravo, Giangiacomo & Farjam, Mike & Grimaldo Moreno, Francisco & Birukou, Aliaksandr & Squazzoni, Flaminio, 2018. "Hidden connections: Network effects on editorial decisions in four computer science journals," Journal of Informetrics, Elsevier, vol. 12(1), pages 101-112.
    2. Ofori, Isaac K. & Gbolonyo, Emmanuel Y. & Ojong, Nathanael, 2023. "Foreign direct investment and inclusive green growth in Africa: Energy efficiency contingencies and thresholds," Energy Economics, Elsevier, vol. 117(C).
    3. Li, Heyang & Wu, Meijun & Wang, Yougui & Zeng, An, 2022. "Bibliographic coupling networks reveal the advantage of diversification in scientific projects," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Maroua Afi & Jay Parsons, 2023. "Integrated vs. Specialized Farming Systems for Sustainable Food Production: Comparative Analysis of Systems’ Technical Efficiency in Nebraska," Sustainability, MDPI, vol. 15(6), pages 1-17, March.
    5. Gralka, Sabine, 2018. "Stochastic frontier analysis in higher education: A systematic review," CEPIE Working Papers 05/18, Technische Universität Dresden, Center of Public and International Economics (CEPIE).
    6. Stefano Mainardi, 2021. "Parametric and Semiparametric Efficiency Frontiers in Fishery Analysis: Overview and Case Study on the Falkland Islands," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 79(2), pages 169-210, June.
    7. Jean Pierre Doussoulin & Cristian Colther, 2022. "Evaluating the Efficiency of Municipal Solid Waste Collection Services in Developing Countries: The Case of Chile," Sustainability, MDPI, vol. 14(23), pages 1-19, November.
    8. Yang, Zhenbing & Chen, Zhuo & Shao, Shuai & Yang, Lili, 2022. "Unintended consequences of additional support on the publications of universities: Evidence from China," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    9. Ofori, Isaac K. & Gbolonyo, Emmanuel Y. & Ojong, Nathanael, 2022. "Foreign Direct Investment and Inclusive Green Growth in Africa: Energy Efficiency Contingencies and Thresholds," MPRA Paper 115379, University Library of Munich, Germany, revised 09 Nov 2022.
    10. John A. Jinapor & Shafic Suleman & Richard Stephens Cromwell, 2023. "Energy Consumption and Environmental Quality in Africa: Does Energy Efficiency Make Any Difference?," Sustainability, MDPI, vol. 15(3), pages 1-26, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    2. Barra, Cristian & Lagravinese, Raffaele & Zotti, Roberto, 2018. "Does econometric methodology matter to rank universities? An analysis of Italian higher education system," Socio-Economic Planning Sciences, Elsevier, vol. 62(C), pages 104-120.
    3. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    4. Gralka, Sabine, 2018. "Stochastic frontier analysis in higher education: A systematic review," CEPIE Working Papers 05/18, Technische Universität Dresden, Center of Public and International Economics (CEPIE).
    5. Calogero Guccio & Marco Ferdinando Martorana & Isidoro Mazza, 2016. "Efficiency assessment and convergence in teaching and research in Italian public universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1063-1094, June.
    6. Laureti, Tiziana & Secondi, Luca & Biggeri, Luigi, 2014. "Measuring the efficiency of teaching activities in Italian universities: An information theoretic approach," Economics of Education Review, Elsevier, vol. 42(C), pages 147-164.
    7. Liu, John S. & Lu, Louis Y.Y. & Lu, Wen-Min, 2016. "Research fronts in data envelopment analysis," Omega, Elsevier, vol. 58(C), pages 33-45.
    8. Moncayo–Martínez, Luis A. & Ramírez–Nafarrate, Adrián & Hernández–Balderrama, María Guadalupe, 2020. "Evaluation of public HEI on teaching, research, and knowledge dissemination by Data Envelopment Analysis," Socio-Economic Planning Sciences, Elsevier, vol. 69(C).
    9. Manuel Salas-Velasco, 2020. "Measuring and explaining the production efficiency of Spanish universities using a non-parametric approach and a bootstrapped-truncated regression," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 825-846, February.
    10. Kamil Makie{l}a & B{l}a.zej Mazur, 2020. "Stochastic Frontier Analysis with Generalized Errors: inference, model comparison and averaging," Papers 2003.07150, arXiv.org, revised Oct 2020.
    11. Chen, Zhuo & Yang, Zhenbing & Yang, Lili, 2020. "How to optimize the allocation of research resources? An empirical study based on output and substitution elasticities of universities in Chinese provincial level," Socio-Economic Planning Sciences, Elsevier, vol. 69(C).
    12. Cuccia, Tiziana & Guccio, Calogero & Rizzo, Ilde, 2016. "The effects of UNESCO World Heritage List inscription on tourism destinations performance in Italian regions," Economic Modelling, Elsevier, vol. 53(C), pages 494-508.
    13. Lampe, Hannes W. & Hilgers, Dennis, 2015. "Trajectories of efficiency measurement: A bibliometric analysis of DEA and SFA," European Journal of Operational Research, Elsevier, vol. 240(1), pages 1-21.
    14. Mehdi Rhaiem, 2017. "Measurement and determinants of academic research efficiency: a systematic review of the evidence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 581-615, February.
    15. Sickles, Robin C. & Song, Wonho & Zelenyuk, Valentin, 2018. "Econometric Analysis of Productivity: Theory and Implementation in R," Working Papers 18-008, Rice University, Department of Economics.
    16. Orea, Luis, 2019. "The Econometric Measurement of Firms’ Efficiency," Efficiency Series Papers 2019/02, University of Oviedo, Department of Economics, Oviedo Efficiency Group (OEG).
    17. Agasisti, Tommaso & Barra, Cristian & Zotti, Roberto, 2016. "Evaluating the efficiency of Italian public universities (2008–2011) in presence of (unobserved) heterogeneity," Socio-Economic Planning Sciences, Elsevier, vol. 55(C), pages 47-58.
    18. Bellio, Ruggero & Grassetti, Luca, 2011. "Semiparametric stochastic frontier models for clustered data," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 71-83, January.
    19. Tommaso Agasisti & Giuseppe Munda & Ralph Hippe, 2019. "Measuring the efficiency of European education systems by combining Data Envelopment Analysis and Multiple-Criteria Evaluation," Journal of Productivity Analysis, Springer, vol. 51(2), pages 105-124, June.
    20. Calogero Guccio & Marco Ferdinando Martorana & Luisa Monaco, 2016. "Evaluating the impact of the Bologna Process on the efficiency convergence of Italian universities: a non-parametric frontier approach," Journal of Productivity Analysis, Springer, vol. 45(3), pages 275-298, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:11:y:2017:i:3:p:613-628. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.