IDEAS home Printed from https://ideas.repec.org/p/wai/econwp/10-03.html
   My bibliography  Save this paper

The Merits of Using Citation-Based Journal Weighting Schemes to Measure Research Performance in Economics: The Case of New Zealand

Author

Listed:

Abstract

In this study we test various citation-based journal weighting schemes, especially those based on the Liebowitz and Palmer methodology, as to their suitability for use in a nation- wide research funding model. Using data generated by New Zealand’s academic economists, we compare the performance of departments, and individuals, under each of our selected schemes; and we then proceed to contrast these results with those generated by direct citation counts. Our findings suggest that if all citations are deemed to be of equal value, then schemes based on the Liebowitz and Palmer methodology yield problematic outcomes. We also demonstrate that even between weighting schemes based on a common methodology, major differences are found to exist in departmental and individual outcomes.

Suggested Citation

  • David L. Anderson & John Tressler, 2010. "The Merits of Using Citation-Based Journal Weighting Schemes to Measure Research Performance in Economics: The Case of New Zealand," Working Papers in Economics 10/03, University of Waikato.
    • Handle: RePEc:wai:econwp:10/03
    as

    Download full text from publisher

    File URL: https://repec.its.waikato.ac.nz/wai/econwp/1003.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kodrzycki Yolanda K. & Yu Pingkang, 2006. "New Approaches to Ranking Economics Journals," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(1), pages 1-44, August.
    2. Wall Howard J, 2009. "Don't Get Skewed Over by Journal Rankings," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 9(1), pages 1-12, August.
    3. Magnus Henrekson & Daniel Waldenström, 2011. "How Should Research Performance Be Measured? A Study Of Swedish Economists," Manchester School, University of Manchester, vol. 79(6), pages 1139-1156, December.
    4. David L. Anderson & John Tressler, 2009. "The Excellence in Research for Australia Scheme: An Evaluation of the Draft Journal Weights for Economics," Working Papers in Economics 09/07, University of Waikato.
    5. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    6. Davis, Paul & Papanek, Gustav F, 1984. "Faculty Ratings of Major Economics Departments by Citations," American Economic Review, American Economic Association, vol. 74(1), pages 225-230, March.
    7. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    8. Bush, Winston C & Hamelman, Paul W & Staaf, Robert J, 1974. "A Quality Index for Economic Journals," The Review of Economics and Statistics, MIT Press, vol. 56(1), pages 123-125, February.
    9. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    10. Graves, Philip E & Marchand, James R & Thompson, Randal, 1982. "Economics Departmental Rankings: Research Incentives, Constraints, and Efficiency," American Economic Review, American Economic Association, vol. 72(5), pages 1131-1141, December.
    11. Hirsch, Barry T, et al, 1984. "Economics Departmental Rankings: Comment [Economics Departmental Rankings: Research Incentives, Constraints, and Efficiency]," American Economic Review, American Economic Association, vol. 74(4), pages 822-826, September.
    12. David L. Anderson & John Tressler, 2009. "The ‘Excellence in Research for Australia’ Scheme: A Test Drive of Draft Journal Weights with New Zealand Data," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 16(4), pages 7-24.
    13. Liebowitz, S J & Palmer, J P, 1984. "Assessing the Relative Impacts of Economic Journals," Journal of Economic Literature, American Economic Association, vol. 22(1), pages 77-88, March.
    14. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    15. John Gibson, 2000. "Research productivity in New Zealand university economics departments: Comment and update," New Zealand Economic Papers, Taylor & Francis Journals, vol. 34(1), pages 73-87.
    16. Kristie M. Engemann & Howard J. Wall, 2009. "A journal ranking for the ambitious economist," Review, Federal Reserve Bank of St. Louis, vol. 91(May), pages 127-140.
    17. Joseph Macri & Dipendra Sinha, 2006. "Rankings Methodology for International Comparisons of Institutions and Individuals: an Application to Economics in Australia and New Zealand," Journal of Economic Surveys, Wiley Blackwell, vol. 20(1), pages 111-156, February.
    18. Rombouts, Jeroen V. K. & Bauwens, Luc, 2004. "Econometrics," Papers 2004,33, Humboldt University of Berlin, Center for Applied Statistics and Economics (CASE).
      • BAUWENS, Luc & ROMBOUTS, Jeroen V.K., 2004. "Econometrics," LIDAM Reprints CORE 1713, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    19. Tom Coupé, 2003. "Revealed Performances: Worldwide Rankings of Economists and Economics Departments, 1990-2000," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1309-1345, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John Tressler & David L. Anderson, 2012. "Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 19(1), pages 17-40.
    2. David L. Anderson & John Tressler, 2011. "The Merits of Using Citations to Measure Research Output in Economics Departments: The New Zealand Case," Working Papers in Economics 11/11, University of Waikato.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    2. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    3. David Anderson & John Tressler, 2008. "Research output in New Zealand economics departments 2000-2006: A stock approach," New Zealand Economic Papers, Taylor & Francis Journals, vol. 42(2), pages 155-189.
    4. Joseph Macri & Dipendra Sinha, 2006. "Rankings Methodology for International Comparisons of Institutions and Individuals: an Application to Economics in Australia and New Zealand," Journal of Economic Surveys, Wiley Blackwell, vol. 20(1), pages 111-156, February.
    5. Seiler, Christian & Wohlrabe, Klaus, 2012. "Ranking economists on the basis of many indicators: An alternative approach using RePEc data," Journal of Informetrics, Elsevier, vol. 6(3), pages 389-402.
    6. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    7. Joseph Macri & Michael McAleer & Dipendra Sinha, 2009. "On the Robustness of Alternative Rankings Methodologies For Australian and New Zealand Economics Departments," CIRJE F-Series CIRJE-F-660, CIRJE, Faculty of Economics, University of Tokyo.
    8. David L. Anderson & John Tressler, 2011. "The Merits of Using Citations to Measure Research Output in Economics Departments: The New Zealand Case," Working Papers in Economics 11/11, University of Waikato.
    9. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    10. Howard J. Wall, 2009. "Journal rankings in economics: handle with care," Working Papers 2009-014, Federal Reserve Bank of St. Louis.
    11. Yu, Xiaohua & Gao, Zhifeng, 2010. "An updated ranking of the economic research institutions in China (2000-2009)," China Economic Review, Elsevier, vol. 21(4), pages 571-581, December.
    12. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    13. David L. Anderson & John Tressler, 2009. "The Excellence in Research for Australia Scheme: An Evaluation of the Draft Journal Weights for Economics," Working Papers in Economics 09/07, University of Waikato.
    14. John Gibson & David L. Anderson & John Tressler, 2017. "Citations Or Journal Quality: Which Is Rewarded More In The Academic Labor Market?," Economic Inquiry, Western Economic Association International, vol. 55(4), pages 1945-1965, October.
    15. John Tressler & David L. Anderson, 2012. "Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'," Agenda - A Journal of Policy Analysis and Reform, Australian National University, College of Business and Economics, School of Economics, vol. 19(1), pages 17-40.
    16. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    17. David L. Anderson & John Tressler, 2011. "Ranking Economics Departments In Terms Of Residual Productivity: New Zealand Economics Departments, 2000–2006," Australian Economic Papers, Wiley Blackwell, vol. 50(4), pages 157-168, December.
    18. David L. Anderson & John Tressler, 2008. "Research Output in New Zealand Economics Department 2000-2006," Working Papers in Economics 08/05, University of Waikato.
    19. Joseph Macri & Michael McAleer & Dipendra Sinha, 2010. "On the robustness of alternative rankings methodologies: Australian and New Zealand economics departments, 1988 to 2002," Applied Economics, Taylor & Francis Journals, vol. 42(10), pages 1257-1268.
    20. Javier Ruiz-Castillo, 2008. "Economics research in Spain during the 1990s: a literature review," Spanish Economic Review, Springer;Spanish Economic Association, vol. 10(3), pages 221-249, September.

    More about this item

    Keywords

    economics departments; research output; citations; impact factors;
    All these keywords.

    JEL classification:

    • A19 - General Economics and Teaching - - General Economics - - - Other
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wai:econwp:10/03. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Geua Boe-Gibson (email available below). General contact details of provider: https://edirc.repec.org/data/dewaknz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.