IDEAS home Printed from https://ideas.repec.org/p/vuw/vuwcpf/21105.html
   My bibliography  Save this paper

Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes

Author

Listed:
  • Buckle, Robert A.
  • Creedy, John
  • Ball, Ashley

Abstract

This paper examines the transformation of New Zealand universities following the introduction in 2003 of the Performance-Based Research Fund System (PBRFS), which assesses performance quality using a peer-review process, and allocates funds based on individual researcher performance. The analysis, based on a social accounting framework, utilises longitudinal researcher data available from the three full assessment rounds, in 2003, 2012 and 2018. The longitudinal data enable identification of entry, exit and quality transformation of researchers and their contribution to changes in university and discipline research quality. The dynamics are found to be closely related to the new incentives created by the assessment system According to the quality metric used by the PBRFS, the research quality of NZ universities increased substantially over the period, although the rate of increase was much slower during the second period, 2012 to 2018, and considerable heterogeneity across universities and disciplines was revealed. Much of the improvement can be attributed to the high exit rate of lower-quality researchers. New entrants consistently reduced the average quality of all groups, reflecting the difficulty of recruiting high-quality researchers. Changes in the discipline composition of universities made a negligible contribution compared to improvements in the quality of the stock of researchers.

Suggested Citation

  • Buckle, Robert A. & Creedy, John & Ball, Ashley, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Working Paper Series 21105, Victoria University of Wellington, Chair in Public Finance.
  • Handle: RePEc:vuw:vuwcpf:21105
    as

    Download full text from publisher

    File URL: https://ir.wgtn.ac.nz/handle/123456789/21105
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:clg:wpaper:2009-04 is not listed on IDEAS
    2. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    3. Robert A. Buckle & John Creedy, 2019. "The evolution of research quality in New Zealand universities as measured by the performance-based research fund process," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(2), pages 144-165, May.
    4. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    5. Robert A. Buckle & John Creedy, 2019. "An evaluation of metrics used by the Performance-based Research Fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(3), pages 270-287, September.
    6. Ehrenberg, Ronald & Kasper, Hirschel & Rees, Daniel, 1991. "Faculty turnover at American colleges and universities: Analyses of AAUP data," Economics of Education Review, Elsevier, vol. 10(2), pages 99-110, June.
    7. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    8. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    9. Anthony Shorrocks, 2013. "Decomposition procedures for distributional analysis: a unified framework based on the Shapley value," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 11(1), pages 99-126, March.
    10. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    11. A. Abigail Payne & Joanne Roberts, 2010. "Government Oversight of Public Universities: Are Centralized Performance Schemes Related to Increased Quantity or Quality?," The Review of Economics and Statistics, MIT Press, vol. 92(1), pages 207-212, February.
    12. Peter Woelert & Lachlan McKenzie, 2018. "Follow the money? How Australian universities replicate national performance-based funding mechanisms," Research Evaluation, Oxford University Press, vol. 27(3), pages 184-195.
    13. Ben R Martin, 2011. "The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster?," Research Evaluation, Oxford University Press, vol. 20(3), pages 247-254, September.
    14. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    15. Jung Cheol Shin & William K. Cummings, 2010. "Multilevel analysis of academic publishing across disciplines: research preference, collaboration, and time on research," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 581-594, November.
    16. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    17. Boyle Glenn, 2008. "Pay Peanuts and Get Monkeys? Evidence from Academia," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 8(1), pages 1-26, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    2. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    3. Buckle, Robert A & Creedy, John, 2022. "The Performance Based Research Fund in NZ: Taking Stock and Looking Forward," Working Paper Series 21354, Victoria University of Wellington, Chair in Public Finance.
    4. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 21104, Victoria University of Wellington, Chair in Public Finance.
    2. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    3. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    4. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    5. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    6. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    7. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    8. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    9. Robert A. Buckle & John Creedy, 2020. "The ‘disciplinary effect’ of the performance-based research fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 54(2), pages 107-126, May.
    10. Buckle, Robert A & Creedy, John, 2022. "The Performance Based Research Fund in NZ: Taking Stock and Looking Forward," Working Paper Series 21354, Victoria University of Wellington, Chair in Public Finance.
    11. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    12. Mike Thelwall & Kayvan Kousha & Emma Stuart & Meiko Makita & Mahshid Abdoli & Paul Wilson & Jonathan Levitt, 2023. "In which fields are citations indicators of research quality?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(8), pages 941-953, August.
    13. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    14. Mehmet Pinar, 2023. "Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5731-5766, October.
    15. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    16. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    17. Degl’Innocenti, Marta & Matousek, Roman & Tzeremes, Nickolaos G., 2019. "The interconnections of academic research and universities’ “third mission”: Evidence from the UK," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    18. John Gibson, 2021. "The micro‐geography of academic research: How distinctive is economics?," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(4), pages 467-484, September.
    19. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    20. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vuw:vuwcpf:21105. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Library Technology Services (email available below). General contact details of provider: https://edirc.repec.org/data/fcvuwnz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.