IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v52y2023i6s0048733323000641.html
   My bibliography  Save this article

Performance-based research funding: Evidence from the largest natural experiment worldwide

Author

Listed:
  • Banal-Estañol, Albert
  • Jofre-Bonet, Mireia
  • Iori, Giulia
  • Maynou, Laia
  • Tumminello, Michele
  • Vassallo, Pietro

Abstract

A performance-based research funding system (PRFS) is a nationwide incentive scheme that promotes and rewards university research performance through competition for government funding. The UK’s PRFS, currently the Research Excellence Framework (REF), is considered the oldest, largest and most developed payment-by-results system in academia worldwide. Surprisingly, and despite the strong criticisms, little has been done to quantitatively and casually evaluate the intended and unintended effects of the PRFSs. In this paper, we evaluate the incremental impact of the REF 2014 in the fields of Economics and Business. We use a synthetic control method to compare the performance of UK universities with their artificial counterfactual units constructed using data from US universities. Our analysis shows, on the whole, that the introduction of the REF had a significant and positive impact on the quantity and quality of the scientific research produced at UK universities. However, we do not find a significant effect on the per author measures, suggesting that the REF did not result in an increase in research productivity. We also show that the effects are more heterogeneous across universities than across academic disciplines. We do not find evidence of a shift of research focus from Economics to Business topics, as some feared. But our analysis indicates that the REF 2014 may have contributed to the concentration of research excellence in elite institutions.

Suggested Citation

  • Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    • Handle: RePEc:eee:respol:v:52:y:2023:i:6:s0048733323000641
    DOI: 10.1016/j.respol.2023.104780
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733323000641
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2023.104780?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robert A. Buckle & John Creedy & Norman Gemmell, 2022. "Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3021-3047, June.
    2. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    3. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the research excellence framework using departmental h-index: revisited," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 1013-1017, September.
    4. Koenraad Debackere & Wolfgang Glänzel, 2004. "Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(2), pages 253-276, February.
    5. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    6. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    7. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the Research Excellence Framework using departmental h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2165-2180, March.
    8. Bruno S. Frey & Reto Jegen, 2001. "Motivation Crowding Theory," Journal of Economic Surveys, Wiley Blackwell, vol. 15(5), pages 589-611, December.
    9. Koen Jonkers & Thomas Zacharewicz, 2016. "Research Performance Based Funding Systems: a Comparative Assessment," JRC Research Reports JRC101043, Joint Research Centre.
    10. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    11. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    12. Björn Hammarfelt & Sarah de Rijcke, 2015. "Accountability in context: effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University," Research Evaluation, Oxford University Press, vol. 24(1), pages 63-77.
    13. Thomas Zacharewicz & Benedetto Lepori & Emanuela Reale & Koen Jonkers, 2019. "Performance-based research funding in EU Member States—a comparative assessment," Science and Public Policy, Oxford University Press, vol. 46(1), pages 105-115.
    14. Auranen, Otto & Nieminen, Mika, 2010. "University research funding and publication performance--An international comparison," Research Policy, Elsevier, vol. 39(6), pages 822-834, July.
    15. Joonha Jeon & So Young Kim, 2018. "Is the gap widening among universities? On research output inequality and its measurement in the Korean higher education system," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(2), pages 589-606, March.
    16. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    17. Engelbert Stockhammer & Quirin Dammerer & Sukriti Kapur, 2021. "The Research Excellence Framework 2014, journal ratings and the marginalisation of heterodox economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 45(2), pages 243-269.
    18. Basso, Antonella & di Tollo, Giacomo, 2022. "Prediction of UK research excellence framework assessment by the departmental h-index," European Journal of Operational Research, Elsevier, vol. 296(3), pages 1036-1049.
    19. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    20. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    21. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    22. Frederic S. Lee & Xuan Pham & Gyun Gu, 2013. "The UK Research Assessment Exercise and the narrowing of UK economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 37(4), pages 693-717.
    23. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    24. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ajab Khan & Ali Sina Önder & Sercan Özcan, 2023. "Does Performance-based Public Funding Pay off? UK’s Research Excellence Framework (REF) and Research Productivity," Working Papers in Economics & Finance 2023-08, University of Portsmouth, Portsmouth Business School, Economics and Finance Subject Group.
    2. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    3. Tomasz Serwach, 2023. "The European Union and within‐country income inequalities. The case of the new member states," The World Economy, Wiley Blackwell, vol. 46(7), pages 1890-1939, July.
    4. Tomasz Serwach, 2022. "The European Union and within-country income inequalities. The case of the New Member States," Working Papers hal-03548416, HAL.
    5. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    6. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    7. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2021. "Decoupling synthetic control methods to ensure stability, accuracy and meaningfulness," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 12(4), pages 549-584, December.
    8. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    9. Olper, Alessandro & Curzi, Daniele & Swinnen, Johan, 2018. "Trade liberalization and child mortality: A Synthetic Control Method," World Development, Elsevier, vol. 110(C), pages 394-410.
    10. Mehmet Pinar, 2023. "Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5731-5766, October.
    11. Matej Opatrny, 2021. "The impact of the Brexit vote on UK financial markets: a synthetic control method approach," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 48(2), pages 559-587, May.
    12. Emery, Thomas & Mélon, Lela & Spruk, Rok, 2023. "Does e-procurement matter for economic growth? Subnational evidence from Australia," The Quarterly Review of Economics and Finance, Elsevier, vol. 89(C), pages 318-334.
    13. Joseph Cummins & Brock Smith & Douglas L. Miller & David Eliot Simon, 2023. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Working papers 2023-07, University of Connecticut, Department of Economics.
    14. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    15. Echevarría, Cruz A. & Hasancebi, Serhat & García-Enríquez, Javier, 2022. "Economic Effects of Macao’s Integration with Mainland China: A Causal Inference Study," Journal of Economic Integration, Center for Economic Integration, Sejong University, vol. 37(2), pages 179-215.
    16. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.
    17. Manuel Funke & Moritz Schularick & Christoph Trebesch, 2023. "Populist Leaders and the Economy," American Economic Review, American Economic Association, vol. 113(12), pages 3249-3288, December.
    18. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    19. Camilla Beck Olsen & Hans Olav Melberg, 2018. "Did adolescents in Norway respond to the elimination of copayments for general practitioner services?," Health Economics, John Wiley & Sons, Ltd., vol. 27(7), pages 1120-1130, July.
    20. Silvia Marchesi & Tania Masi, 2019. "Sovereign risk after sovereign restructuring. Private and official default," Working Papers 423, University of Milano-Bicocca, Department of Economics, revised Nov 2019.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:52:y:2023:i:6:s0048733323000641. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.