IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp11198.html
   My bibliography  Save this paper

Rising Stars

Author

Listed:
  • Battistin, Erich

    () (University of Maryland)

  • Ovidi, Marco

    () (Queen Mary, University of London)

Abstract

We use the UK's 2014 Research Excellence Framework (REF) to study which attributes characterize a top-scoring (four-star) publication in Economics and Econometrics. We frame the analysis as a classification problem and, using information in official documents, derive conditions to infer the unobservable score that panellists awarded to each publication. Juxtaposing institutions' submissions with REF outcomes provides information on the latent pass-marks used for assigning quality levels, which respond to journal prestige measured by the Thomson Reuters Article Influence Score. We explore this statistical feature in the econometric analysis, which reveals the limited contribution to awarded quality made by other publication attributes, possibly unobservable to us, conditional on the Article Influence Score. We conclude that, in large-scale and costly evaluations such as the REF, the time-consuming task of peer reviews should be devoted to publications not in academic outlets with unambiguously top-scoring bibliometric indicators of journal impact. Our model also predicts a ranking of academic journals consistent with the classification of REF panellists.

Suggested Citation

  • Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
  • Handle: RePEc:iza:izadps:dp11198
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp11198.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. De Fraja, Gianni & Facchini, Giovanni & Gathergood, John, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," CEPR Discussion Papers 11638, C.E.P.R. Discussion Papers.
    2. Arne Risa Hole, 2017. "Ranking Economics Journals Using Data From a National Research Evaluation Exercise," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 621-636, October.
    3. Sofronis Clerides & Panos Pashardes & Alexandros Polycarpou, 2011. "Peer Review vs Metric‐based Assessment: Testing for Bias in the RAE Ratings of UK Economics Departments," Economica, London School of Economics and Political Science, vol. 78(311), pages 565-583, July.
    4. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    5. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    6. Natalia Zinovyeva & Manuel Bagues, 2015. "The Role of Connections in Academic Promotions," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 264-292, April.
    7. Sauer, Raymond D, 1988. "Estimates of the Returns to Quality and Coauthorship in Economic Academia," Journal of Political Economy, University of Chicago Press, vol. 96(4), pages 855-866, August.
    8. David Card & Stefano DellaVigna, 2017. "What do Editors Maximize? Evidence from Four Leading Economics Journals," NBER Working Papers 23282, National Bureau of Economic Research, Inc.
    9. Erich Battistin & Barbara Sianesi, 2011. "Misclassified Treatment Status and Treatment Effects: An Application to Returns to Education in the United Kingdom," The Review of Economics and Statistics, MIT Press, vol. 93(2), pages 495-509, May.
    10. David N. Laband, 2013. "On the Use and Abuse of Economics Journal Rankings," Economic Journal, Royal Economic Society, vol. 0, pages 223-254, August.
    11. Glenn Ellison, 2002. "The Slowdown of the Economics Publishing Process," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 947-993, October.
    12. Joshua Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2017. "Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship," NBER Working Papers 23698, National Bureau of Economic Research, Inc.
    13. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    14. repec:spr:scient:v:113:y:2017:i:2:d:10.1007_s11192-017-2519-3 is not listed on IDEAS
    15. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    16. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    17. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    18. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    19. John Hudson, 2013. "Ranking Journals," Economic Journal, Royal Economic Society, vol. 0, pages 202-222, August.
    20. Pierre Régibeau & Katharine E. Rockett, 2016. "Research assessment and recognized excellence: simple bibliometrics for more efficient academic research evaluations," Economic Policy, CEPR;CES;MSH, vol. 31(88), pages 611-652.
    21. Tombazos, Christis G. & Dobra, Matthew, 2014. "Formulating research policy on expert advice," European Economic Review, Elsevier, vol. 72(C), pages 166-181.
    22. Laband, David N & Piette, Michael J, 1994. "The Relative Impacts of Economics Journals: 1970-1990," Journal of Economic Literature, American Economic Association, vol. 32(2), pages 640-666, June.
    23. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    24. Liebowitz, S J & Palmer, J P, 1984. "Assessing the Relative Impacts of Economic Journals," Journal of Economic Literature, American Economic Association, vol. 22(1), pages 77-88, March.
    25. Colussi, Tommaso, 2015. "Social Ties in Academia: A Friend is a Treasure," IZA Discussion Papers 9414, Institute of Labor Economics (IZA).
    26. Francesco Bartolucci & Valentino Dardanoni & Franco Peracchi, 2015. "Ranking scientific journals via latent class models for polytomous item response data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 1025-1049, October.
    27. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    education policy; higher education; journal ranking; research funding;

    JEL classification:

    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • H52 - Public Economics - - National Government Expenditures and Related Policies - - - Government Expenditures and Education
    • H83 - Public Economics - - Miscellaneous Issues - - - Public Administration
    • H83 - Public Economics - - Miscellaneous Issues - - - Public Administration
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I28 - Health, Education, and Welfare - - Education - - - Government Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp11198. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Holger Hinte). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.