IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i1d10.1007_s11192-020-03643-x.html
   My bibliography  Save this article

Are Italian research assessment exercises size-biased?

Author

Listed:
  • Camil Demetrescu

    (Sapienza University of Rome)

  • Andrea Ribichini

    (Sapienza University of Rome)

  • Marco Schaerf

    (Sapienza University of Rome)

Abstract

Research assessment exercises have enjoyed ever-increasing popularity in many countries in recent years, both as a method to guide public funds allocation and as a validation tool for adopted research support policies. Italy’s most recently completed evaluation effort (VQR 2011–14) required each university to submit to the Ministry for Education, University, and Research (MIUR) 2 research products per author (3 in the case of other research institutions), chosen in such a way that the same product is not assigned to two authors belonging to the same institution. This constraint suggests that larger institutions, where collaborations among colleagues may be more frequent, could suffer a size-related bias in their evaluation scores. To validate our claim, we investigate the outcome of artificially splitting Sapienza University of Rome, one of the largest universities in Europe, in a number of separate partitions, according to several criteria, noting significant score increases for several partitioning scenarios.

Suggested Citation

  • Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2020. "Are Italian research assessment exercises size-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 533-549, October.
  • Handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03643-x
    DOI: 10.1007/s11192-020-03643-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03643-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-020-03643-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    2. Jiro Nemoto & Noriko Furumatsu, 2014. "Scale and scope economies of Japanese private universities revisited with an input distance function approach," Journal of Productivity Analysis, Springer, vol. 41(2), pages 213-226, April.
    3. repec:adr:anecst:y:1998:i:49-50:p:05 is not listed on IDEAS
    4. Graziella Bertocchi & Alfonso Gambardella & Tullio Jappelli & Carmela Anna Nappi & Franco Peracchi, 2016. "Comment to: Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 349-353, July.
    5. Avkiran, Necmi K., 2001. "Investigating technical and scale efficiencies of Australian Universities through data envelopment analysis," Socio-Economic Planning Sciences, Elsevier, vol. 35(1), pages 57-80, March.
    6. Calabrese, Armando & Capece, Guendalina & Costa, Roberta & Di Pillo, Francesca & Giuffrida, Stefania, 2018. "A ‘power law’ based method to reduce size-related bias in indicators of knowledge performance: An application to university research assessment," Journal of Informetrics, Elsevier, vol. 12(4), pages 1263-1281.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    8. Alessio Ancaiani & Alberto F. Anfossi & Anna Barbara & Sergio Benedetto & Brigida Blasi & Valentina Carletti & Tindaro Cicero & Alberto Ciolfi & Filippo Costa & Giovanna Colizza & Marco Costantini & F, 2015. "Evaluating scientific research in Italy: The 2004–10 research evaluation exercise," Research Evaluation, Oxford University Press, vol. 24(3), pages 242-255.
    9. Koshal, Rajindar K. & Koshal, Manjulika, 1999. "Economies of scale and scope in higher education: a case of comprehensive universities," Economics of Education Review, Elsevier, vol. 18(2), pages 269-277, April.
    10. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Murgia, Gianluca, 2013. "The collaboration behaviors of scientists in Italy: A field level analysis," Journal of Informetrics, Elsevier, vol. 7(2), pages 442-454.
    11. Golden, John & Carstensen, Fred V., 1992. "Academic research productivity, department size and organization: Further results, comment," Economics of Education Review, Elsevier, vol. 11(2), pages 153-160, June.
    12. Golden, John & Carstensen, Fred V., 1992. "Academic research productivity, department size and organization: Further results, rejoinder," Economics of Education Review, Elsevier, vol. 11(2), pages 169-171, June.
    13. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    14. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    15. Alberto Baccini & Giuseppe De Nicolao, 2017. "A letter on Ancaiani et al. ‘Evaluating scientific research in Italy: the 2004-10 research evaluation exercise’," Research Evaluation, Oxford University Press, vol. 26(4), pages 353-357.
    16. Worthington, Andrew C. & Higgs, Helen, 2014. "Economies of scale and scope in Australian urban water utilities," Utilities Policy, Elsevier, vol. 31(C), pages 52-62.
    17. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2019. "A gender analysis of top scientists’ collaboration behavior: evidence from Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 405-418, August.
    18. Bozeman, Barry & Gaughan, Monica, 2011. "How do men and women differ in research collaborations? An analysis of the collaborative motives and strategies of academic researchers," Research Policy, Elsevier, vol. 40(10), pages 1393-1402.
    19. Vincent Larivière & Yves Gingras & Éric Archambault, 2006. "Canadian collaboration networks: A comparative analysis of the natural sciences, social sciences and the humanities," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 519-533, September.
    20. Jordan, John M. & Meador, Mark & Walters, Stephen J. K., 1989. "Academic research productivity, department size and organization: Further results," Economics of Education Review, Elsevier, vol. 8(4), pages 345-352, August.
    21. Keiji Hashimoto & Elchanan Cohn, 1997. "Economies of Scale and Scope in Japanese Private Universities," Education Economics, Taylor & Francis Journals, vol. 5(2), pages 107-115.
    22. Izadi, Hooshang & Johnes, Geraint & Oskrochi, Reza & Crouchley, Robert, 2002. "Stochastic frontier estimation of a CES cost function: the case of higher education in Britain," Economics of Education Review, Elsevier, vol. 21(1), pages 63-71, February.
    23. Alberto Baccini & Giuseppe Nicolao, 2016. "Reply to the comment of Bertocchi et al," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1675-1684, September.
    24. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    25. Abbott, M. & Doucouliagos, C., 2003. "The efficiency of Australian universities: a data envelopment analysis," Economics of Education Review, Elsevier, vol. 22(1), pages 89-97, February.
    26. James D. Adams & Zvi Griliches, 1998. "Research Productivity in a System of Universities," Annals of Economics and Statistics, GENES, issue 49-50, pages 127-162.
    27. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: the effects of changing the rules of the game during the game," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(1), pages 229-238, July.
    28. Andrea Bonaccorsi & Cinzia Daraio, 2005. "Exploring size and agglomeration effects on public research productivity," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(1), pages 87-120, March.
    29. Jordan, John M. & Meador, Mark & Walters, Stephen J. K., 1988. "Effects of department size and organization on the research productivity of academic economists," Economics of Education Review, Elsevier, vol. 7(2), pages 251-255, April.
    30. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    31. Sergio Benedetto & Tindaro Cicero & Marco Malgarini & Carmen Nappi, 2017. "Reply to the letter on Ancaiani et al. ‘Evaluating Scientific research in Italy: The 2004–10 research evaluation exercise’," Research Evaluation, Oxford University Press, vol. 26(4), pages 358-360.
    32. David N. Laband & Bernard F. Lentz, 2003. "New Estimates of Economies of Scale and Scope in Higher Education," Southern Economic Journal, John Wiley & Sons, vol. 70(1), pages 172-183, July.
    33. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Murgia, Gianluca, 2013. "Gender differences in research collaboration," Journal of Informetrics, Elsevier, vol. 7(4), pages 811-822.
    34. R. Kenna & B. Berche, 2011. "Critical mass and the dependency of research quality on group size," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 527-540, February.
    35. Longlong, Hou & Fengliang, Li & Weifang, Min, 2009. "Multi-product total cost functions for higher education: The case of Chinese research universities," Economics of Education Review, Elsevier, vol. 28(4), pages 505-511, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Silvia Bacci & Bruno Bertaccini & Alessandra Petrucci, 2023. "Insights from the co-authorship network of the Italian academic statisticians," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4269-4303, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Efficiency and economies of scale and specialization in European universities: A directional distance approach," Journal of Informetrics, Elsevier, vol. 9(3), pages 430-448.
    2. Bonaccorsi, Andrea & Belingheri, Paola & Secondi, Luca, 2021. "The research productivity of universities. A multilevel and multidisciplinary analysis on European institutions," Journal of Informetrics, Elsevier, vol. 15(2).
    3. Andrea Bonaccorsi & Luca Secondi, 2017. "The determinants of research performance in European universities: a large scale multilevel analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1147-1178, September.
    4. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    5. Liang-Cheng Zhang & Andrew C. Worthington, 2018. "Explaining Estimated Economies of Scale and Scope in Higher Education: A Meta-Regression Analysis," Research in Higher Education, Springer;Association for Institutional Research, vol. 59(2), pages 156-173, March.
    6. Andrea Bonaccorsi & Cinzia Daraio & Leopold Simar, 2014. "Efficiency and economies of scale and scope in European universities. A directional distance approach," DIAG Technical Reports 2014-08, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    7. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    8. Shamohammadi, Mehdi & Oh, Dong-hyun, 2019. "Measuring the efficiency changes of private universities of Korea: A two-stage network data envelopment analysis," Technological Forecasting and Social Change, Elsevier, vol. 148(C).
    9. Mehdi Rhaiem, 2017. "Measurement and determinants of academic research efficiency: a systematic review of the evidence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 581-615, February.
    10. Marvin A. Titus & Adriana Vamosiu & Kevin R. McClure, 2017. "Are Public Master’s Institutions Cost Efficient? A Stochastic Frontier and Spatial Analysis," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(5), pages 469-496, August.
    11. Joanna Wolszczak-Derlacz & Aleksandra Parteka, 2011. "Efficiency of European public higher education institutions: a two-stage multicountry approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 887-917, December.
    12. Tasso Brandt & Torben Schubert, 2013. "Is the university model an organizational necessity? Scale and agglomeration effects in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 541-565, February.
    13. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    14. Lee, Boon L. & Worthington, Andrew C., 2016. "A network DEA quantity and quality-orientated production model: An application to Australian university research services," Omega, Elsevier, vol. 60(C), pages 26-33.
    15. Torben Schubert & Guoliang Yang, 2016. "Institutional change and the optimal size of universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1129-1153, September.
    16. Maria Olivares & Heike Wetzel, 2014. "Editor's Choice Competing in the Higher Education Market: Empirical Evidence for Economies of Scale and Scope in German Higher Education Institutions," CESifo Economic Studies, CESifo Group, vol. 60(4), pages 653-680.
    17. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2016. "The north–south divide in the Italian higher education system," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2093-2117, December.
    18. Schuelke-Leech, Beth-Anne, 2013. "Resources and research: An empirical study of the influence of departmental research resources on individual STEM researchers involvement with industry," Research Policy, Elsevier, vol. 42(9), pages 1667-1678.
    19. Fieger, Peter & Villano, Renato & Cooksey, Ray, 2015. "Two dimensional efficiency measurements in Australian TAFE Institutes," MPRA Paper 64625, University Library of Munich, Germany.
    20. Torben Schubert, 2014. "Are there scale economies in scientific production? On the topic of locally increasing returns to scale," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(2), pages 393-408, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03643-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.