IDEAS home Printed from https://ideas.repec.org/a/bla/econom/v88y2021i352p1107-1129.html
   My bibliography  Save this article

Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm

Author

Listed:
  • Daniele Checchi
  • Alberto Ciolfi
  • Gianni De Fraja
  • Irene Mazzotta
  • Stefano Verzillo

Abstract

This paper determines the assessment of publications submitted to the UK research evaluation carried out in 2014, the REF, which would have resulted if they had been assessed with the bibliometric algorithm used by the Italian evaluation agency, ANVUR, for its evaluation of the research of Italian universities. We find extremely high correlations between the two assessment approaches.

Suggested Citation

  • Daniele Checchi & Alberto Ciolfi & Gianni De Fraja & Irene Mazzotta & Stefano Verzillo, 2021. "Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm," Economica, London School of Economics and Political Science, vol. 88(352), pages 1107-1129, October.
    • Handle: RePEc:bla:econom:v:88:y:2021:i:352:p:1107-1129
    DOI: 10.1111/ecca.12373
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ecca.12373
    Download Restriction: no
    File URL: https://libkey.io/10.1111/ecca.12373?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. De Fraja, Gianni & Facchini, Giovanni & Gathergood, John, 2016. "How Much Is That Star in the Window? Professorial Salaries and Research Performance in UK Universities," CEPR Discussion Papers 11638, C.E.P.R. Discussion Papers.
    2. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    3. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the Research Excellence Framework using departmental h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2165-2180, March.
    4. J. E. Hirsch, 2010. "An index to quantify an individual’s scientific research output that takes into account the effect of multiple coauthorship," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 741-754, December.
    5. Gianni De Fraja & Giovanni Facchini & John Gathergood, 2019. "Academic salaries and public evaluation of university research: Evidence from the UK Research Excellence Framework," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 34(99), pages 523-583.
    6. Graziella Bertocchi & Alfonso Gambardella & Tullio Jappelli & Carmela Anna Nappi & Franco Peracchi, 2016. "Comment to: Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 349-353, July.
    7. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    8. Alberto Anfossi & Alberto Ciolfi & Filippo Costa & Giorgio Parisi & Sergio Benedetto, 2016. "Large-scale assessment of research outputs through a weighted combination of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 671-683, May.
    9. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the research excellence framework using departmental h-index: revisited," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 1013-1017, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hadavand, Aboozar & Hamermesh, Daniel S. & Wilson, Wesley W., 2021. "Publishing Economics: How Slow? Why Slow? Is Slow Productive? Fixing Slow?," IZA Discussion Papers 14643, Institute of Labor Economics (IZA).
    2. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    3. Antonin Mac'e, 2017. "The Limits of Citation Counts," Papers 1711.02695, arXiv.org, revised Sep 2023.
    4. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    5. Hamermesh, Daniel S. & Kosnik, Lea-Rachel, 2022. "Aging in Style: Does How We Write Matter?," IZA Discussion Papers 15739, Institute of Labor Economics (IZA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    2. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    3. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    4. Alberto Baccini & Giuseppe Nicolao, 2016. "Reply to the comment of Bertocchi et al," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1675-1684, September.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    6. Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2020. "Are Italian research assessment exercises size-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 533-549, October.
    7. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    8. Shahd Al-Janabi & Lee Wei Lim & Luca Aquili, 2021. "Development of a tool to accurately predict UK REF funding allocation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8049-8062, September.
    9. Erich Battistin & Marco Ovidi, 2022. "Rising Stars: Expert Reviews and Reputational Yardsticks in the Research Excellence Framework," Economica, London School of Economics and Political Science, vol. 89(356), pages 830-848, October.
    10. James Tooley & Barrie Craven, 2018. "Private Sector Alternatives to the Research Excellence Framework for University League Tables," Economic Affairs, Wiley Blackwell, vol. 38(3), pages 434-443, October.
    11. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    13. Battistin, Erich & Ovidi, Marco, 2017. "Rising Stars," IZA Discussion Papers 11198, Institute of Labor Economics (IZA).
    14. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    15. Graziella Bertocchi & Alfonso Gambardella & Tullio Jappelli & Carmela Anna Nappi & Franco Peracchi, 2016. "Comment to: Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 349-353, July.
    16. Basso, Antonella & di Tollo, Giacomo, 2022. "Prediction of UK research excellence framework assessment by the departmental h-index," European Journal of Operational Research, Elsevier, vol. 296(3), pages 1036-1049.
    17. Banal-Estañol, Albert & Jofre-Bonet, Mireia & Iori, Giulia & Maynou, Laia & Tumminello, Michele & Vassallo, Pietro, 2023. "Performance-based research funding: Evidence from the largest natural experiment worldwide," Research Policy, Elsevier, vol. 52(6).
    18. Stephan B. Bruns & David I. Stern, 2016. "Research assessment using early citation information," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 917-935, August.
    19. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    20. Aliakbar Akbaritabar & Niccolò Casnici & Flaminio Squazzoni, 2018. "The conundrum of research productivity: a study on sociologists in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 859-882, March.

    More about this item

    JEL classification:

    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • I28 - Health, Education, and Welfare - - Education - - - Government Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:econom:v:88:y:2021:i:352:p:1107-1129. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.