IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11198.html
   My bibliography  Save this paper

Rising Stars

Author

Listed:
  • Battistin, Erich

    (University of Maryland)

  • Ovidi, Marco

    (Catholic University Milan)

Abstract

We use the UK's 2014 Research Excellence Framework (REF) to study which attributes characterize a top-scoring (four-star) publication in Economics and Econometrics. We frame the analysis as a classification problem and, using information in official documents, derive conditions to infer the unobservable score that panellists awarded to each publication. Juxtaposing institutions' submissions with REF outcomes provides information on the latent pass-marks used for assigning quality levels, which respond to journal prestige measured by the Thomson Reuters Article Influence Score. We explore this statistical feature in the econometric analysis, which reveals the limited contribution to awarded quality made by other publication attributes, possibly unobservable to us, conditional on the Article Influence Score. We conclude that, in large-scale and costly evaluations such as the REF, the time-consuming task of peer reviews should be devoted to publications not in academic outlets with unambiguously top-scoring bibliometric indicators of journal impact. Our model also predicts a ranking of academic journals consistent with the classification of REF panellists.

Suggested Citation

  • Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp11198
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp11198.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Sofronis Clerides & Panos Pashardes & Alexandros Polycarpou, 2006. "RAE Ratings and Research Quality: The Case of Economics Departments," University of Cyprus Working Papers in Economics 7-2006, University of Cyprus Department of Economics.
    2. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    5. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    6. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," Discussion Papers 2016-13, University of Nottingham, GEP.
    7. Colussi, Tommaso, 2015. "Social Ties in Academia: A Friend is a Treasure," IZA Discussion Papers 9414, Institute of Labor Economics (IZA).
    8. Francesco Bartolucci & Valentino Dardanoni & Franco Peracchi, 2015. "Ranking scientific journals via latent class models for polytomous item response data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 1025-1049, October.
    9. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    10. Jill Johnes & Jim Taylor & Brian Francis, 1993. "The Research Performance of UK Universities: A Statistical Analysis of the Results of the 1989 Research Selectivity Exercise," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 156(2), pages 271-286, March.
    11. Natalia Zinovyeva & Manuel Bagues, 2015. "The Role of Connections in Academic Promotions," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 264-292, April.
    12. David N. Laband, 2013. "On the Use and Abuse of Economics Journal Rankings," Economic Journal, Royal Economic Society, vol. 0, pages 223-254, August.
    13. Glenn Ellison, 2002. "The Slowdown of the Economics Publishing Process," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 947-993, October.
    14. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    15. John Hudson, 2013. "Ranking Journals," Economic Journal, Royal Economic Society, vol. 0, pages 202-222, August.
    16. Pierre Régibeau & Katharine E. Rockett, 2016. "Research assessment and recognized excellence: simple bibliometrics for more efficient academic research evaluations," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 31(88), pages 611-652.
    17. Tombazos, Christis G. & Dobra, Matthew, 2014. "Formulating research policy on expert advice," European Economic Review, Elsevier, vol. 72(C), pages 166-181.
    18. Sauer, Raymond D, 1988. "Estimates of the Returns to Quality and Coauthorship in Economic Academia," Journal of Political Economy, University of Chicago Press, vol. 96(4), pages 855-866, August.
    19. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    20. Laband, David N & Piette, Michael J, 1994. "The Relative Impacts of Economics Journals: 1970-1990," Journal of Economic Literature, American Economic Association, vol. 32(2), pages 640-666, June.
    21. Sofronis Clerides & Panos Pashardes & Alexandros Polycarpou, 2011. "Peer Review vs Metric‐based Assessment: Testing for Bias in the RAE Ratings of UK Economics Departments," Economica, London School of Economics and Political Science, vol. 78(311), pages 565-583, July.
    22. Liebowitz, S J & Palmer, J P, 1984. "Assessing the Relative Impacts of Economic Journals," Journal of Economic Literature, American Economic Association, vol. 22(1), pages 77-88, March.
    23. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    24. David Card & Stefano DellaVigna, 2017. "What do Editors Maximize? Evidence from Four Leading Economics Journals," NBER Working Papers 23282, National Bureau of Economic Research, Inc.
    25. Erich Battistin & Barbara Sianesi, 2011. "Misclassified Treatment Status and Treatment Effects: An Application to Returns to Education in the United Kingdom," The Review of Economics and Statistics, MIT Press, vol. 93(2), pages 495-509, May.
    26. Joshua Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2017. "Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship," NBER Working Papers 23698, National Bureau of Economic Research, Inc.
    27. John Hudson, 2017. "Identifying economics’ place amongst academic disciplines: a science or a social science?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 735-750, November.
    28. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    29. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Checchi, Daniele & Fenizia, Alessandra & Lucifora, Claudio, 2021. "Public Sector Jobs: Working in the Public Sector in Europe and the US," IZA Discussion Papers 14514, Institute of Labor Economics (IZA).
    2. Basiglio Stefania & Foresta Alessandra & Turati Gilberto, 2021. "Impatience and crime. Evidence from the NLSY97," Working papers 073, Department of Economics and Statistics (Dipartimento di Scienze Economico-Sociali e Matematico-Statistiche), University of Torino.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    2. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    3. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    4. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    5. Chagas , André Luis Squarize, 2017. "Publish or Perish: um ranking de revistas da subárea de Economia Regional e Urbana para os pesquisadores brasileiros," Revista Brasileira de Estudos Regionais e Urbanos, Associação Brasileira de Estudos Regionais e Urbanos (ABER), vol. 11(4), pages 515-536.
    6. Ham, John C. & Wright, Julian & Ye, Ziqiu, 2023. "Documenting and Explaining the Dramatic Rise of the New Society Journals in Economics," IZA Discussion Papers 16337, Institute of Labor Economics (IZA).
    7. Tombazos, Christis G. & Dobra, Matthew, 2014. "Formulating research policy on expert advice," European Economic Review, Elsevier, vol. 72(C), pages 166-181.
    8. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    9. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    10. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & J. Martinez-Baena, 2012. "On first quartile journals which are not of highest impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 925-943, March.
    11. Waltman, Ludo & van Eck, Nees Jan, 2008. "Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie," Journal of Informetrics, Elsevier, vol. 2(4), pages 369-372.
    12. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2011. "Overall prestige of journals with ranking score above a given threshold," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 229-243, October.
    13. László Á. Kóczy & Alexandru Nichifor & Martin Strobel, 2010. "Intellectual Infl uence: Quality versus Quantity," Working Paper Series 1009, Óbuda University, Keleti Faculty of Business and Management.
    14. Kóczy, L.Á. & Nichifor, A. & Strobel, M., 2009. "Article length bias in journal rankings," Research Memorandum 016, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    15. László Á. Kóczy & Martin Strobel, 2010. "The World Cup of Economics Journals: A Ranking by a Tournament Method," Working Paper Series 1011, Óbuda University, Keleti Faculty of Business and Management.
    16. repec:esx:essedp:757 is not listed on IDEAS
    17. Azar, Ofer H., 2007. "Behavioral economics and socio-economics journals: A citation-based ranking," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 36(3), pages 451-462, June.
    18. László Kóczy & Alexandru Nichifor, 2013. "The intellectual influence of economic journals: quality versus quantity," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 52(3), pages 863-884, April.
    19. Ignacio Palacios-Huerta & Oscar Volij, 2002. "The Measure of Intellectual Influence," Working Papers 2002-13, Brown University, Department of Economics.
    20. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    21. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.

    More about this item

    Keywords

    research funding; journal ranking; higher education; education policy;
    All these keywords.

    JEL classification:

    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • H83 - Public Economics - - Miscellaneous Issues - - - Public Administration
    • H83 - Public Economics - - Miscellaneous Issues - - - Public Administration
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I28 - Health, Education, and Welfare - - Education - - - Government Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11198. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.