IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_6818.html
   My bibliography  Save this paper

Publication Performance vs. Influence: On the Questionable Value of Quality Weighted Publication Rankings

Author

Listed:
  • Justus Haucap
  • Tobias Thomas
  • Klaus Wohlrabe

Abstract

In broad parts of the scientific community the position in publication performance rankings, based on journal quality ratings is seen as highly reputational for the scientist. This contribution provides evidence that, at least in economics, such publication performance measures can not always be reconciled with measures for academic influence such as citation-based measures. We analyze data from the Scopus database as well as from the prestigious German-based Handelsblatt ranking for 100 renowned economists (lifetime achievement). Scholarly influence is proxied by various bibliometric indicators such as the number of citations, the h-index, the citations of the most cited paper as well as the hardly honorable Pi-Beta-score (“Publications Ignored, By Even The Author(s)”). We argue that publication performance measures based on journal ratings, such as the Handelsblatt rankings, are not good proxies for an economist’s impact within the scientific community. From this perspective the value of publication performance rankings based on journal quality ratings is questionable.

Suggested Citation

  • Justus Haucap & Tobias Thomas & Klaus Wohlrabe, 2017. "Publication Performance vs. Influence: On the Questionable Value of Quality Weighted Publication Rankings," CESifo Working Paper Series 6818, CESifo.
  • Handle: RePEc:ces:ceswps:_6818
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp6818.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. KRAPF, Matthias & SCHLÄPFER, Jörg, 2012. "How Nobel Laureates Would Perform In The Handelsblatt Ranking," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    2. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9, pages 473-505, November.
    3. Bruno S. Frey & Katja Rost, 2010. "Do rankings reflect research quality?," Journal of Applied Economics, Universidad del CEMA, vol. 13, pages 1-38, May.
    4. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    5. Justus Haucap & Tobias Thomas, 2014. "Wissenschaftliche Politikberatung: Erreicht der Rat von Ökonomen Politik und Öffentlichkeit?," Wirtschaftsdienst, Springer;ZBW - Leibniz Information Centre for Economics, vol. 94(3), pages 180-186, March.
    6. Chia-Lin Chang & Michael McAleer & Les Oxley, 2011. "Great Expectatrics: Great Papers, Great Journals, Great Econometrics," Econometric Reviews, Taylor & Francis Journals, vol. 30(6), pages 583-619.
    7. Birkmaier, Daniel & Wohlrabe, Klaus, 2014. "The Matthew effect in economics reconsidered," Journal of Informetrics, Elsevier, vol. 8(4), pages 880-889.
    8. Philippe Mongeon & Adèle Paul-Hus, 2016. "The journal coverage of Web of Science and Scopus: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 213-228, January.
    9. Pierre-Philippe Combes & Laurent Linnemer, 2010. "Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics," Working Papers halshs-00520325, HAL.
    10. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    11. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    12. Bernd Fitzenberger & Ute Schulze, 2014. "Up or Out: Research Incentives and Career Prospects of Postdocs in Germany," German Economic Review, Verein für Socialpolitik, vol. 15(2), pages 287-328, May.
    13. Schulze Günther G. & Wiermann Christian & Warning Susanne, 2008. "What and How Long Does It Take to Get Tenure? The Case of Economics and Business Administration in Austria, Germany and Switzerland," German Economic Review, De Gruyter, vol. 9(4), pages 473-505, December.
    14. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    15. Sturm Jan-Egbert & Ursprung Heinrich W., 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, De Gruyter, vol. 18(4), pages 492-515, December.
    16. Felix Schläpfer & Friedrich Schneider, 2010. "Messung der akademischen Forschungsleistung in den Wirtschaftswissenschaften: Reputation vs. Zitierhäufigkeiten," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 11(4), pages 325-339, November.
    17. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    18. Michael Graber & Andrey Launov & Klaus Wälde, 2008. "Publish or Perish? The Increasing Importance of Publications for Prospective Economics Professors in Austria, Germany and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 457-472, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Syed Hasan & Robert Breunig, 2021. "Article length and citation outcomes," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7583-7608, September.
    2. Joseph Gerald Hirschberg & Jeanette Ngaire Lye, 2020. "Grading Journals In Economics: The Abcs Of The Abdc," Journal of Economic Surveys, Wiley Blackwell, vol. 34(4), pages 876-921, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Butz & Klaus Wohlrabe, 2016. "Die Ökonomen-Rankings 2015 von Handelsblatt, FAZ und RePEc: Methodik, Ergebnisse, Kritik und Vergleich," ifo Working Paper Series 212, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    2. Alexander Butz & Klaus Wohlrabe, 2016. "Comments and Criticism of the 2015 Economists Rankings of Handelsblatt, the FAZ and RePEc," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 69(10), pages 37-44, May.
    3. Bräuninger Michael & Haucap Justus & Muck Johannes, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 12(4), pages 339-371, November.
    4. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2017. "A Meta-Ranking of Economics Publications," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 70(16), pages 40-42, August.
    5. Klaus Wohlrabe, 2018. "Selected Remarks on Frankfurter Allgemeine Zeitung Economists’ Ranking 2018," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 71(20), pages 29-33, October.
    6. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    7. Berlemann, Michael & Haucap, Justus, 2012. "Which factors drive the decision to boycott and opt out of research rankings? A note," DICE Discussion Papers 72, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    8. Michael Berlemann & Justus Haucap, 2012. "Which Factors Drive the Decision to Boycott and Opt Out of Research Rankings?," CESifo Working Paper Series 3997, CESifo.
    9. Bräuninger, Michael & Haucap, Justus & Muck, Johannes, 2011. "Was lesen und schätzen Ökonomen im Jahr 2011?," DICE Ordnungspolitische Perspektiven 18, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    10. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    11. Klaus Wohlrabe, 2020. "Some Remarks on the FAZ Economist Ranking 2020," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 73(11), pages 53-57, November.
    12. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    13. Ali Sina Önder & Sascha Schweitzer, 2017. "Catching up or falling behind? Promising changes and persistent patterns across cohorts of economics PhDs in German-speaking countries from 1991 to 2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1297-1331, March.
    14. Libman, A., 2010. "Economics in Germany – from National to Global," Journal of the New Economic Association, New Economic Association, issue 8, pages 155-158.
    15. KRAPF, Matthias & SCHLÄPFER, Jörg, 2012. "How Nobel Laureates Would Perform In The Handelsblatt Ranking," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    16. Rolf Ketzler & Klaus F. Zimmermann, 2013. "A citation-analysis of economic research institutes," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 1095-1112, June.
    17. Julia Muschallik & Kerstin Pull, 2016. "Mentoring in higher education: does it enhance mentees’ research productivity?," Education Economics, Taylor & Francis Journals, vol. 24(2), pages 210-223, April.
    18. Jan-Egbert Sturm & Heinrich W. Ursprung, 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, Verein für Socialpolitik, vol. 18(4), pages 492-515, November.
    19. Bäker, Agnes, 2015. "Non-tenured post-doctoral researchers’ job mobility and research output: An analysis of the role of research discipline, department size, and coauthors," Research Policy, Elsevier, vol. 44(3), pages 634-650.
    20. Justus Haucap, 2020. "Wirtschaftswissenschaftliche Politikberatung in Deutschland: Stärken, Schwächen, Optimierungspotenzial," Springer Books, in: Dirk Loerwald (ed.), Ökonomische Erkenntnisse verständlich vermitteln, pages 45-78, Springer.

    More about this item

    Keywords

    economics; academic reputation; academic rankings; influence; citations; Scopus; Handelsblatt ranking; academic journals;
    All these keywords.

    JEL classification:

    • A12 - General Economics and Teaching - - General Economics - - - Relation of Economics to Other Disciplines
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_6818. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.