IDEAS home Printed from https://ideas.repec.org/p/wai/econwp/12-03.html
   My bibliography  Save this paper

Evaluating Research - Peer Review Team Assessment and Journal-Based Bibliographic Measures: New Zealand PBRF Research Output Scores in 2006

Author

Listed:
  • David L. Anderson

    (Queen's University)

  • Warren Smart

    (Ministry of Education)

  • John Tressler

    (University of Waikato)

Abstract

This paper concerns the relationship between the assessment of the research of individual academics by peer or expert review teams with a variety of bibliometric schemes based on journal quality weights. Specifically, for a common group of economists from New Zealand departments of economics the relationship between Performance-Based Research Fund (PBRF) Research Output measures for those submitting new research portfolios in 2006 are compared with evaluations of journal based research over the 2000-2005 assessment period. This comparison identifies the journal weighting schemes that appear most similar to PBRF peer evaluations. The paper provides an indication of the ‘power or aggressiveness’ of PBRF evaluations in terms of the weighting given to quality. The implied views of PBRF peer review teams are also useful in assessing common assumptions made in evaluating journal based research.

Suggested Citation

  • David L. Anderson & Warren Smart & John Tressler, 2012. "Evaluating Research - Peer Review Team Assessment and Journal-Based Bibliographic Measures: New Zealand PBRF Research Output Scores in 2006," Working Papers in Economics 12/03, University of Waikato.
    • Handle: RePEc:wai:econwp:12/03
    as

    Download full text from publisher

    File URL: https://repec.its.waikato.ac.nz/wai/econwp/1203.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    2. David L. Anderson & John Tressler, 2011. "Ranking Economics Departments In Terms Of Residual Productivity: New Zealand Economics Departments, 2000–2006," Australian Economic Papers, Wiley Blackwell, vol. 50(4), pages 157-168, December.
    3. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    2. Uddin, Shahadat & Khan, Arif, 2016. "The impact of author-selected keywords on citation counts," Journal of Informetrics, Elsevier, vol. 10(4), pages 1166-1177.
    3. David L. Anderson & John Tressler, 2013. "The New Zealand performance-based research fund and its impact on publication activity in economics," Research Evaluation, Oxford University Press, vol. 23(1), pages 1-11, September.
    4. Robert A. Buckle & John Creedy, 2019. "The evolution of research quality in New Zealand universities as measured by the performance-based research fund process," New Zealand Economic Papers, Taylor & Francis Journals, vol. 53(2), pages 144-165, May.
    5. Thelwall, Mike & Wilson, Paul, 2014. "Regression for citation data: An evaluation of different methods," Journal of Informetrics, Elsevier, vol. 8(4), pages 963-971.
    6. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 21104, Victoria University of Wellington, Chair in Public Finance.
    7. Robert A. Buckle and John Creedy, 2018. "The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 24(1), pages 25-48.
    8. John Gibson & David L. Anderson & John Tressler, 2017. "Citations Or Journal Quality: Which Is Rewarded More In The Academic Labor Market?," Economic Inquiry, Western Economic Association International, vol. 55(4), pages 1945-1965, October.
    9. David L. Anderson & John Tressler, 2015. "Are Researcher Rankings Stable Across Alternative Output Measurement Schemes in the Context of a Time Limited Research Evaluation? The New Zealand Case," Working Papers in Economics 15/10, University of Waikato.
    10. Imad A. Moosa, 2016. "A Critique of the Bucket Classification of Journals: The ABDC List as an Example," The Economic Record, The Economic Society of Australia, vol. 92(298), pages 448-463, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Gibson & David L. Anderson & John Tressler, 2017. "Citations Or Journal Quality: Which Is Rewarded More In The Academic Labor Market?," Economic Inquiry, Western Economic Association International, vol. 55(4), pages 1945-1965, October.
    2. Martina Cioni & Giovanni Federico & Michelangelo Vasta, 2020. "The long-term evolution of economic history: evidence from the top five field journals (1927–2017)," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 14(1), pages 1-39, January.
    3. Konstantinos Eleftheriou & Patroklos Patsoulis & Michael Polemis, 2023. "Convergence among academic journals in accounting: a note," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(2), pages 1055-1069, February.
    4. Martina Cioni & Govanni Federico & Michelangelo Vasta, 2018. "Ninety years of publications in Economic History: evidence from the top five field journals (1927-2017)," Department of Economics University of Siena 791, Department of Economics, University of Siena.
    5. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    6. Gibson, John, 2014. "Returns to articles versus pages in academic publishing: Do salary-setters show ‘article illusion’?," Economics Letters, Elsevier, vol. 125(3), pages 343-346.
    7. Eleftheriou, Konstantinos & Polemis, Michael, 2019. "Convergence or divergence in finance journal ranking?," MPRA Paper 93528, University Library of Munich, Germany.
    8. Eleftheriou, Konstantinos & Polemis, Michael, 2020. "One list to fit them all: What do we learn from journal ranking?," Finance Research Letters, Elsevier, vol. 35(C).
    9. Claudio Vitari, 2014. "Electronic currencies for purposive degrowth?," Working paper serie RMT - Grenoble Ecole de Management hal-00975432, HAL.
    10. Laurent Bouton, 2011. "Good rankings are bad - Why reliable rankings can hurt consumers," Boston University - Department of Economics - Working Papers Series WP2011-002, Boston University - Department of Economics.
    11. Clément Bosquet & Pierre‐Philippe Combes & Cecilia García‐Peñalosa, 2019. "Gender and Promotions: Evidence from Academic Economists in France," Scandinavian Journal of Economics, Wiley Blackwell, vol. 121(3), pages 1020-1053, July.
    12. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    13. A. Muravyev., 2013. "On Scientific Value of Russian Journals in Economics and Related Fields," VOPROSY ECONOMIKI, N.P. Redaktsiya zhurnala "Voprosy Economiki", vol. 4.
    14. Bosquet, Clément & Combes, Pierre-Philippe & Garcia-Penalosa, Cecilia, 2013. "Gender and competition: evidence from academic promotions in France," LSE Research Online Documents on Economics 58350, London School of Economics and Political Science, LSE Library.
    15. Howard, Peter H. & Sylvan, Derek, 2015. "The Economic Climate: Establishing Consensus on the Economics of Climate Change," 2015 AAEA & WAEA Joint Annual Meeting, July 26-28, San Francisco, California 205761, Agricultural and Applied Economics Association.
    16. Paul Johnson & Chris Papageorgiou, 2020. "What Remains of Cross-Country Convergence?," Journal of Economic Literature, American Economic Association, vol. 58(1), pages 129-175, March.
    17. Xingwei Hu, 2020. "Sorting Big Data by Revealed Preference with Application to College Ranking," Papers 2003.12198, arXiv.org.
    18. Di Vaio, Gianfranco & Waldenström, Daniel & Weisdorf, Jacob, 2012. "Citation success: Evidence from economic history journal publications," Explorations in Economic History, Elsevier, vol. 49(1), pages 92-104.
    19. Matthias Aistleitner & Jakob Kapeller & Stefan Steinerberger, 2018. "Citation Patterns in Economics and Beyond," Working Papers Series 85, Institute for New Economic Thinking.
    20. Faria João R & Mixon Franklin G & Upadhyaya Kamal P, 2018. "Editor Reputation and Journal Quality: The Case of Regional Economic Association Journals," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 18(1), pages 1-20, January.

    More about this item

    Keywords

    bibliometrics; journal weighting schemes; PBRF; research measurement;
    All these keywords.

    JEL classification:

    • A19 - General Economics and Teaching - - General Economics - - - Other
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wai:econwp:12/03. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Geua Boe-Gibson (email available below). General contact details of provider: https://edirc.repec.org/data/dewaknz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.