IDEAS home Printed from https://ideas.repec.org/a/taf/nzecpp/v53y2019i3p270-287.html

An evaluation of metrics used by the Performance-based Research Fund process in New Zealand

Author

Listed:
  • Robert A. Buckle
  • John Creedy

Abstract

The New Zealand Performance-based Research Fund applies a set of metrics to assess researchers and rank disciplines and universities. The process involves giving a total weighted score (based on three components) to individuals and then assigning them to one of four Quality Categories (QCs), used to derive Average Quality Scores (AQS). This paper evaluates the properties of these metrics and argues that QC thresholds influence the final distribution of scores. The paper also demonstrates that the derivation of AQSs depends on the weights assigned to each QC and the distribution of portfolios. The method used to determine raw scores also has an independent effect on the distribution of scores. The paper compares how research rankings of New Zealand universities would vary if alternative summary measures, based on the total weighted scores rather than QCs, were used to evaluate performance.

Suggested Citation

  • Robert A. Buckle & John Creedy, 2019. "An evaluation of metrics used by the Performance-based Research Fund process in New Zealand," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(3), pages 270-287, September.
    • Handle: RePEc:taf:nzecpp:v:53:y:2019:i:3:p:270-287
    DOI: 10.1080/00779954.2018.1480054
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/00779954.2018.1480054
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/00779954.2018.1480054?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or

    for a different version of it.

    Other versions of this item:

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    2. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    3. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    4. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    5. Robert A. Buckle and John Creedy, 2018. "The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 24(1), pages 25-48.
    6. Robert A. Buckle & John Creedy & Norman Gemmell, 2020. "Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand," Applied Economics, Taylor & Francis Journals, vol. 52(36), pages 3919-3932, July.
    7. Robert A. Buckle & John Creedy & Ashley Ball, 2021. "Fifteen Years of a PBRFS in New Zealand: Incentives and Outcomes," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 54(2), pages 208-230, June.
    8. Buckle, Robert A & Creedy, John, 2022. "The Performance Based Research Fund in NZ: Taking Stock and Looking Forward," Working Paper Series 21354, Victoria University of Wellington, Chair in Public Finance.
    9. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:nzecpp:v:53:y:2019:i:3:p:270-287. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: http://www.tandfonline.com/RNZP20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.