IDEAS home Printed from https://ideas.repec.org/p/knz/dpteco/1002.html
   My bibliography  Save this paper

Research evaluation and journal quality weights: Much ado about nothing?

Author

Listed:
  • Matthias Krapf

    (Department of Economics, University of Konstanz, Germany)

Abstract

Research evaluations based on quality weighted publication output are often criticized on account of the employed journal quality weights. This study shows that evaluations of entire research organizations are very robust with respect to the choice of readily available weighting schemes. We document this robustness by applying rather different weighting schemes to otherwise identical rankings. Our unit of analysis consists of German, Austrian and Swiss university departments in business administration and economics.

Suggested Citation

  • Matthias Krapf, 2010. "Research evaluation and journal quality weights: Much ado about nothing?," Working Paper Series of the Department of Economics, University of Konstanz 2010-02, Department of Economics, University of Konstanz.
    • Handle: RePEc:knz:dpteco:1002
    as

    Download full text from publisher

    File URL: http://www.uni-konstanz.de/FuF/wiwi/workingpaperseries/WP_Krapf-2-10.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Oliver Fabel & Miriam Hein & Robert Hofmeister, 2008. "Research Productivity in Business Economics: An Investigation of Austrian, German and Swiss Universities," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 506-531, November.
    2. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    3. Klaus Ritzberger, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, Verein für Socialpolitik, vol. 9, pages 402-430, November.
    4. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "Zeitschriftenrankings für die Wirtschaftswissenschaften – Konstruktion eines umfassenden Metaindexes," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 286-305, August.
    5. Michael Rauber & Heinrich W. Ursprung, 2008. "Life Cycle and Cohort Productivity in Economic Research: The Case of Germany," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 431-456, November.
    6. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    7. Friedrich Schneider & Heinrich W. Ursprung, 2008. "The 2008 GEA Journal‐Ranking for the Economics Profession," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 532-538, November.
    8. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    9. Friedrich Schneider, 2008. "Special Issue on the Economics of the Economics Profession: Editorial," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 397-401, November.
    10. Friedrich Schneider & Heinrich W. Ursprung, 2008. "The 2008 GEA Journal‐Ranking for the Economics Profession," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 532-538, November.
    11. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    12. Partha, Dasgupta & David, Paul A., 1994. "Toward a new economics of science," Research Policy, Elsevier, vol. 23(5), pages 487-521, September.
    13. repec:jns:jbstat:v:227:y:2007:i:2:p:187-208 is not listed on IDEAS
    14. Pierre-Philippe Combes & Laurent Linnemer, 2003. "Where are the Economists Who Publish? Publication Concentration and Rankings in Europe Based on Cumulative Publications," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1250-1308, December.
    15. Pierre-Philippe Combes & Laurent Linnemer, 2010. "Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics," Working Papers halshs-00520325, HAL.
    16. Friedrich Schneider, 2008. "Special Issue on the Economics of the Economics Profession: Editorial," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 397-401, November.
    17. Ritzberger Klaus, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, De Gruyter, vol. 9(4), pages 402-430, December.
    18. Oliver Fabel & Miriam Hein & Robert Hofmeister, 2008. "Research Productivity in Business Economics: An Investigation of Austrian, German and Swiss Universities," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 506-531, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J. Paul Elhorst & Katarina Zigova, 2011. "Evidence of Competition in Research Activity among Economic Department using Spatial Econometric Techniques," Working Paper Series of the Department of Economics, University of Konstanz 2011-04, Department of Economics, University of Konstanz.
    2. Uschi Backes-Gellner & Agnes Bäker & Kerstin Pull, 2018. "The Opportunity Costs of Becoming a Dean: Does Leadership in Academia Crowd Out Research?," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 70(2), pages 189-208, May.
    3. Klaus Gründler & Niklas Potrafke, 2019. "Ideologically-charged terminology: austerity, fiscal consolidation, and sustainable governance," CESifo Working Paper Series 7613, CESifo.
    4. Adam Ayaita & Kerstin Pull & Uschi Backes-Gellner, 2019. "You get what you ‘pay’ for: academic attention, career incentives and changes in publication portfolios of business and economics researchers," Journal of Business Economics, Springer, vol. 89(3), pages 273-290, April.
    5. Tobias Kollmann & Andreas Kuckertz & Christoph Stöckmann & Patrick Krell, 2012. "Die Entrepreneurshipforschung in Deutschland, Österreich und der Schweiz. Eine Resonanzanalyse," ZfKE – Zeitschrift für KMU und Entrepreneurship, Duncker & Humblot, Berlin, vol. 60(1), pages 53-76.
    6. Julia Muschallik & Kerstin Pull, 2016. "Mentoring in higher education: does it enhance mentees’ research productivity?," Education Economics, Taylor & Francis Journals, vol. 24(2), pages 210-223, April.
    7. Jan-Egbert Sturm & Heinrich W. Ursprung, 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 18(4), pages 492-515, November.
    8. Ioana Alexandra Horodnic & Adriana Zaiţ, 2015. "Motivation and research productivity in a university system undergoing transition," Research Evaluation, Oxford University Press, vol. 24(3), pages 282-292.
    9. Ioana Alexandra HORODNIC, 2014. "Academic Performance: Measurement Methods Used In Socio - Economic Sciences," THE YEARBOOK OF THE "GH. ZANE" INSTITUTE OF ECONOMIC RESEARCHES, Gheorghe Zane Institute for Economic and Social Research ( from THE ROMANIAN ACADEMY, JASSY BRANCH), vol. 23(1), pages 5-17.
    10. Agnes Bäker & Susanne Breuninger & Julia Muschallik & Kerstin Pull & Uschi Backes-Gellner, 2016. "Time to Go? (Inter)National Mobility and Appointment Success of Young Academics," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 17(3), pages 401-421, December.
    11. Kaltrina Nuredini, 2021. "Investigating Altmetric Information For The Top 1000 Journals From Handelsblatt Ranking In Economic And Business Studies," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1315-1343, December.
    12. Matthias Meyer & Rüdiger W. Waldkirch & Michael A. Zaggl, 2012. "Relative Performance Measurement of Researchers: The Impact of Data Source Selection," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 64(4), pages 308-330, October.
    13. Peter Burridge & J. Paul Elhorst & Katarina Zigova, 2016. "Group Interaction in Research and the Use of General Nesting Spatial Models," Advances in Econometrics, in: Spatial Econometrics: Qualitative and Limited Dependent Variables, volume 37, pages 223-258, Emerald Group Publishing Limited.
    14. Guimaraes, Bernardo, 2011. "Qualis as a measuring stick for research output in Economics," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 31(1), March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. Paul Elhorst & Katarina Zigova, 2011. "Evidence of Competition in Research Activity among Economic Department using Spatial Econometric Techniques," Working Paper Series of the Department of Economics, University of Konstanz 2011-04, Department of Economics, University of Konstanz.
    2. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    3. Eduardo A. Haddad & Jesus P. Mena-Chalco, Otávio J.G. Sidone, 2016. "Produção Científica e Redes de Colaboração dos Docentes Vinculados aos Programas de Pós-graduação em Economia no Brasil," Working Papers, Department of Economics 2016_10, University of São Paulo (FEA-USP).
    4. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    5. Chagas , André Luis Squarize, 2017. "Publish or Perish: um ranking de revistas da subárea de Economia Regional e Urbana para os pesquisadores brasileiros," Revista Brasileira de Estudos Regionais e Urbanos, Associação Brasileira de Estudos Regionais e Urbanos (ABER), vol. 11(4), pages 515-536.
    6. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    7. Andreas Haufler & Johannes Rincke, 2009. "Wer trägt bei der Jahrestagung des Vereins für Socialpolitik vor? Eine empirische Analyse," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 10(2), pages 123-145, May.
    8. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    9. John P. Conley & Mario J. Crucini & Robert A. Driskill & Ali Sina Onder, 2011. "Incentives and the Effects of Publication Lags on Life Cycle Research Productivity in Economics," Vanderbilt University Department of Economics Working Papers 1122, Vanderbilt University Department of Economics.
    10. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    11. Jan-Egbert Sturm & Heinrich W. Ursprung, 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 18(4), pages 492-515, November.
    12. Bräuninger Michael & Haucap Justus & Muck Johannes, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 12(4), pages 339-371, November.
    13. Peter Burridge & J. Paul Elhorst & Katarina Zigova, 2016. "Group Interaction in Research and the Use of General Nesting Spatial Models," Advances in Econometrics, in: Spatial Econometrics: Qualitative and Limited Dependent Variables, volume 37, pages 223-258, Emerald Group Publishing Limited.
    14. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9, pages 473-505, November.
    15. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    16. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    17. Ho F. Chan & Franklin G. Mixon & Benno Torgler, 2018. "Relation of early career performance and recognition to the probability of winning the Nobel Prize in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1069-1086, March.
    18. Yu, Xiaohua & Gao, Zhifeng, 2010. "An updated ranking of the economic research institutions in China (2000-2009)," China Economic Review, Elsevier, vol. 21(4), pages 571-581, December.
    19. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    20. Bräuninger, Michael & Haucap, Justus & Muck, Johannes, 2011. "Was lesen und schätzen Ökonomen im Jahr 2011?," DICE Ordnungspolitische Perspektiven 18, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).

    More about this item

    Keywords

    Research evaluation; university management;

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:knz:dpteco:1002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Office Ursprung (email available below). General contact details of provider: https://edirc.repec.org/data/fwkonde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.