IDEAS home Printed from https://ideas.repec.org/p/zbw/dicedp/277.html
   My bibliography  Save this paper

Publication performance vs. influence: On the questionable value of quality weighted publication rankings

Author

Listed:
  • Haucap, Justus
  • Thomas, Tobias
  • Wohlrabe, Klaus

Abstract

In broad parts of the scientific community the position in publication performance rankings, based on journal quality ratings is seen as highly reputational for the scientist. This contribution provides evidence that, at least in economics, such publication performance measures can not always be reconciled with measures for academic influence such as citation-based measures. We analyze data from the Scopus database as well as from the prestigious German-based Handelsblatt ranking for 100 renowned economists (lifetime achievement). Scholarly influence is proxied by various bibliometric indicators such as the number of citations, the h-index, the citations of the most cited paper as well as the hardly honorable Pi-Beta-score ("Publications Ignored, By Even The Author(s)"). We argue that publication performance measures based on journal ratings, such as the Handelsblatt rankings, are not good proxies for an economist's impact within the scientific community. From this perspective the value of publication performance rankings based on journal quality ratings is questionable.

Suggested Citation

  • Haucap, Justus & Thomas, Tobias & Wohlrabe, Klaus, 2017. "Publication performance vs. influence: On the questionable value of quality weighted publication rankings," DICE Discussion Papers 277, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    • Handle: RePEc:zbw:dicedp:277
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/172207/1/1008672254.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Chia-Lin Chang & Michael McAleer & Les Oxley, 2011. "Great Expectatrics: Great Papers, Great Journals, Great Econometrics," Econometric Reviews, Taylor & Francis Journals, vol. 30(6), pages 583-619.
    2. KRAPF, Matthias & SCHLÄPFER, Jörg, 2012. "How Nobel Laureates Would Perform In The Handelsblatt Ranking," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    3. repec:bla:germec:v:9:y:2008:i::p:473-505 is not listed on IDEAS
    4. Bruno S. Frey & Katja Rost, 2010. "Do rankings reflect research quality?," Journal of Applied Economics, Universidad del CEMA, vol. 13, pages 1-38, May.
    5. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    6. Bernd Fitzenberger & Ute Schulze, 2014. "Up or Out: Research Incentives and Career Prospects of Postdocs in Germany," German Economic Review, Verein für Socialpolitik, vol. 15(2), pages 287-328, May.
    7. Schulze Günther G. & Wiermann Christian & Warning Susanne, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, De Gruyter, vol. 9(4), pages 473-505, December.
    8. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    9. Sturm Jan-Egbert & Ursprung Heinrich W., 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, De Gruyter, vol. 18(4), pages 492-515, December.
    10. Justus Haucap & Tobias Thomas, 2014. "Wissenschaftliche Politikberatung: Erreicht der Rat von Ökonomen Politik und Öffentlichkeit?," Wirtschaftsdienst, Springer;ZBW - Leibniz Information Centre for Economics, vol. 94(3), pages 180-186, March.
    11. Birkmaier, Daniel & Wohlrabe, Klaus, 2014. "The Matthew effect in economics reconsidered," Journal of Informetrics, Elsevier, vol. 8(4), pages 880-889.
    12. Felix Schläpfer & Friedrich Schneider, 2010. "Messung der akademischen Forschungsleistung in den Wirtschaftswissenschaften: Reputation vs. Zitierhäufigkeiten," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 11(4), pages 325-339, November.
    13. Philippe Mongeon & Adèle Paul-Hus, 2016. "The journal coverage of Web of Science and Scopus: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 213-228, January.
    14. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    15. Pierre-Philippe Combes & Laurent Linnemer, 2010. "Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics," Working Papers halshs-00520325, HAL.
    16. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    17. repec:bla:germec:v:9:y:2008:i::p:457-472 is not listed on IDEAS
    18. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Joseph Gerald Hirschberg & Jeanette Ngaire Lye, 2020. "Grading Journals In Economics: The Abcs Of The Abdc," Journal of Economic Surveys, Wiley Blackwell, vol. 34(4), pages 876-921, September.
    2. Syed Hasan & Robert Breunig, 2021. "Article length and citation outcomes," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7583-7608, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    2. Alexander Butz & Klaus Wohlrabe, 2016. "Anmerkungen und Kritik zu den Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 69(10), pages 37-44, May.
    3. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    4. Klaus Wohlrabe, 2018. "Einige Anmerkungen zum FAZ-Ökonomenranking 2018," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 71(20), pages 29-33, October.
    5. Michael Bräuninger & Justus Haucap & Johannes Muck, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 12(4), pages 339-371, November.
    6. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2017. "Ein Meta-Ranking volkswirtschaftlicher Fachzeitschriften," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 70(16), pages 40-42, August.
    7. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    8. Klaus Wohlrabe, 2020. "Einige Anmerkungen zum FAZ-Ökonomenranking 2020," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(11), pages 53-57, November.
    9. Berlemann, Michael & Haucap, Justus, 2012. "Which factors drive the decision to boycott and opt out of research rankings? A note," DICE Discussion Papers 72, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    10. Michael Berlemann & Justus Haucap, 2012. "Which Factors Drive the Decision to Boycott and Opt Out of Research Rankings?," CESifo Working Paper Series 3997, CESifo.
    11. Bräuninger, Michael & Haucap, Justus & Muck, Johannes, 2011. "Was lesen und schätzen Ökonomen im Jahr 2011?," DICE Ordnungspolitische Perspektiven 18, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    12. KRAPF, Matthias & SCHLÄPFER, Jörg, 2012. "How Nobel Laureates Would Perform In The Handelsblatt Ranking," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    13. Jan-Egbert Sturm & Heinrich W. Ursprung, 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 18(4), pages 492-515, November.
    14. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    15. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    16. Justus Haucap, 2020. "Wirtschaftswissenschaftliche Politikberatung in Deutschland: Stärken, Schwächen, Optimierungspotenzial," Springer Books, in: Dirk Loerwald (ed.), Ökonomische Erkenntnisse verständlich vermitteln, pages 45-78, Springer.
    17. Rolf Ketzler & Klaus F. Zimmermann, 2013. "A citation-analysis of economic research institutes," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 1095-1112, June.
    18. Ulrich Rendtel, 2019. "Journal-Rankings und Karriere im Fach Statistik an wirtschaftwissenschaftlichen Fakultäten [Journal rankings and career in statistics at economic departments]," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 13(2), pages 125-142, September.
    19. Klaus Gründler & Niklas Potrafke, 2019. "Ideologically-charged terminology: austerity, fiscal consolidation, and sustainable governance," CESifo Working Paper Series 7613, CESifo.
    20. Klaus Wohlrabe, 2011. "Das Handelsblatt- und das RePEc-Ranking im Vergleich," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 64(17), pages 66-71, September.

    More about this item

    Keywords

    economics; academic reputation; academic rankings; influence; citations; Scopus; Handelsblatt ranking; academic journals;
    All these keywords.

    JEL classification:

    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:dicedp:277. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/diduede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.