IDEAS home Printed from https://ideas.repec.org/a/bla/ausecp/v61y2022i3p615-634.html
   My bibliography  Save this article

Methods to evaluate institutional responses to performance‐based research funding systems

Author

Listed:
  • Robert A. Buckle
  • John Creedy

Abstract

Performance‐based research funding systems (PBRFS) have been introduced in many countries for allocating funding to research institutions. There continues to be considerable debate about the effectiveness and consequences of these systems. This paper suggests several methods that can be applied to evaluate how institutions respond to new incentives created by the introduction of a PBRFS. The methods are illustrated using longitudinal data from the New Zealand PBRFS, which assesses institutional performance and allocates funds based on individual researcher performance, although not all methods require individual data. Longitudinal data enable the identification of entry, exit and quality transformation of researchers and the contribution of these dynamics to changes in university and discipline research quality. The approach enables a deeper understanding of individual and institutional responses to PBRFSs, the sustainability of changes and the contributions of changes in researcher quality and discipline composition to changes in institutional performance.

Suggested Citation

  • Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    • Handle: RePEc:bla:ausecp:v:61:y:2022:i:3:p:615-634
    DOI: 10.1111/1467-8454.12263
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/1467-8454.12263
    Download Restriction: no
    File URL: https://libkey.io/10.1111/1467-8454.12263?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    2. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    3. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    4. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    5. repec:clg:wpaper:2009-04 is not listed on IDEAS
    6. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    7. Ehrenberg, Ronald & Kasper, Hirschel & Rees, Daniel, 1991. "Faculty turnover at American colleges and universities: Analyses of AAUP data," Economics of Education Review, Elsevier, vol. 10(2), pages 99-110, June.
    8. Jane Broadbent, 2010. "The UK Research Assessment Exercise: Performance Measurement and Resource Allocation," Australian Accounting Review, CPA Australia, vol. 20(1), pages 14-23, March.
    9. Katharine Barker, 2007. "The UK Research Assessment Exercise: the evolution of a national research evaluation system," Research Evaluation, Oxford University Press, vol. 16(1), pages 3-12, March.
    10. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: a comparison of peer review and bibliometrics rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 929-941, December.
    11. Mattia Cattaneo & Michele Meoli & Andrea Signori, 2016. "Performance-based funding and university research productivity: the moderating effect of university legitimacy," The Journal of Technology Transfer, Springer, vol. 41(1), pages 85-104, February.
    12. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    13. Anthony Shorrocks, 2013. "Decomposition procedures for distributional analysis: a unified framework based on the Shapley value," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 11(1), pages 99-126, March.
    14. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    15. A. Peter W. Hodder & Catherine Hodder, 2010. "Research culture and New Zealand’s performance-based research fund: some insights from bibliographic compilations of research outputs," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 887-901, September.
    16. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    17. Alessio Ancaiani & Alberto F. Anfossi & Anna Barbara & Sergio Benedetto & Brigida Blasi & Valentina Carletti & Tindaro Cicero & Alberto Ciolfi & Filippo Costa & Giovanna Colizza & Marco Costantini & F, 2015. "Evaluating scientific research in Italy: The 2004–10 research evaluation exercise," Research Evaluation, Oxford University Press, vol. 24(3), pages 242-255.
    18. Duncan A Thomas & Maria Nedeva & Mayra M Tirado & Merle Jacob, 2020. "Changing research on research evaluation: A critical literature review to revisit the agenda," Research Evaluation, Oxford University Press, vol. 29(3), pages 275-288.
    19. Castle Jennifer L. & Doornik Jurgen A & Hendry David F., 2011. "Evaluating Automatic Model Selection," Journal of Time Series Econometrics, De Gruyter, vol. 3(1), pages 1-33, February.
    20. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    21. Jenny M. Lewis, 2014. "Research productivity and research system attitudes," Public Money & Management, Taylor & Francis Journals, vol. 34(6), pages 417-424, November.
    22. A. Abigail Payne & Joanne Roberts, 2010. "Government Oversight of Public Universities: Are Centralized Performance Schemes Related to Increased Quantity or Quality?," The Review of Economics and Statistics, MIT Press, vol. 92(1), pages 207-212, February.
    23. Quah, D., 1990. "Galton'S Fallacy And The Tests Of The Convergence Hypothesis," Working papers 552, Massachusetts Institute of Technology (MIT), Department of Economics.
    24. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    25. Peter Woelert & Lachlan McKenzie, 2018. "Follow the money? How Australian universities replicate national performance-based funding mechanisms," Research Evaluation, Oxford University Press, vol. 27(3), pages 184-195.
    26. William J. Moore & Robert J. Newman & Peter J. Sloane & Jeremy D. Steely, 2002. "Productivity Effects of Research Assessment Exercises," Departmental Working Papers 2002-15, Department of Economics, Louisiana State University.
    27. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    28. Koen Jonkers & Thomas Zacharewicz, 2016. "Research Performance Based Funding Systems: a Comparative Assessment," JRC Research Reports JRC101043, Joint Research Centre.
    29. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    30. Jung Cheol Shin & William K. Cummings, 2010. "Multilevel analysis of academic publishing across disciplines: research preference, collaboration, and time on research," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 581-594, November.
    31. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    32. Robert A. Buckle & John Creedy, 2019. "An evaluation of metrics used by the Performance-based Research Fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(3), pages 270-287, September.
    33. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    34. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    35. Boyle Glenn, 2008. "Pay Peanuts and Get Monkeys? Evidence from Academia," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 8(1), pages 1-26, July.
    36. Diana Hicks, 2020. "A superb example of policy evaluation," Research Evaluation, Oxford University Press, vol. 29(3), pages 352-353.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 21104, Victoria University of Wellington, Chair in Public Finance.
    2. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    3. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    4. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    5. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    6. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    7. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    8. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    9. Buckle, Robert A & Creedy, John, 2022. "The Performance Based Research Fund in NZ: Taking Stock and Looking Forward," Working Paper Series 21354, Victoria University of Wellington, Chair in Public Finance.
    10. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    11. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    12. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    13. Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    14. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    15. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    16. Giovanni Abramo & Ciriaco Andrea D’Angelo & Myroslava Hladchenko, 2023. "Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4589-4609, August.
    17. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    18. Dzieżyc, Maciej & Kazienko, Przemysław, 2022. "Effectiveness of research grants funded by European Research Council and Polish National Science Centre," Journal of Informetrics, Elsevier, vol. 16(1).
    19. Mehmet Pinar, 2023. "Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5731-5766, October.
    20. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:ausecp:v:61:y:2022:i:3:p:615-634. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0004-900X .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.