IDEAS home Printed from https://ideas.repec.org/a/bla/jinfst/v66y2015i11p2202-2214.html
   My bibliography  Save this article

The VQR, Italy's second national research assessment: Methodological failures and ranking distortions

Author

Listed:
  • Giovanni Abramo
  • Ciriaco Andrea D'Angelo

Abstract

No abstract is available for this item.

Suggested Citation

  • Giovanni Abramo & Ciriaco Andrea D'Angelo, 2015. "The VQR, Italy's second national research assessment: Methodological failures and ranking distortions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2202-2214, November.
    • Handle: RePEc:bla:jinfst:v:66:y:2015:i:11:p:2202-2214
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/asi.23323
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Frank J. Trueba & Héctor Guerrero, 2004. "A robust formula to credit authors for their publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 181-204, June.
    2. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    3. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    4. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    5. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    6. Lokman I. Meho & Diane H. Sonnenwald, 2000. "Citation ranking versus peer evaluation of senior faculty research performance: A case study of Kurdish scholarship," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(2), pages 123-138.
    7. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Rosati, Francesco, 2013. "The importance of accounting for the number of co-authors and their order when assessing research performance at the individual level in the life sciences," Journal of Informetrics, Elsevier, vol. 7(1), pages 198-208.
    8. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    9. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    10. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    11. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    12. Giovanni Abramo & Ciriaco D’Angelo & Flavia Di Costa & Marco Solazzi, 2011. "The role of information asymmetry in the market for university–industry research collaboration," The Journal of Technology Transfer, Springer, vol. 36(1), pages 84-100, February.
    13. Anthony F. J. van Raan, 2005. "Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 133-143, January.
    14. Peter Weingart, 2005. "Impact of bibliometrics upon the science system: Inadvertent consequences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 117-131, January.
    15. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    16. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "Revisiting the scaling of citations for research assessment," Journal of Informetrics, Elsevier, vol. 6(4), pages 470-479.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Rosati, Francesco, 2016. "A methodology to measure the effectiveness of academic recruitment and turnover," Journal of Informetrics, Elsevier, vol. 10(1), pages 31-42.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    4. Thelwall, Mike, 2016. "The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 110-123.
    5. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    6. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    7. Ellen Hazelkorn & Andrew Gibson, 2017. "Global science, national research, and the question of university rankings," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-11, December.
    8. Domenico A. Maisano & Luca Mastrogiacomo & Fiorenzo Franceschini, 2020. "Short-term effects of non-competitive funding to single academic researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(3), pages 1261-1280, June.
    9. Irene Sala & Maurizio Sobrero, 2021. "Games of policy and practice: multi-level dynamics and the role of universities in knowledge transfer processes," The Journal of Technology Transfer, Springer, vol. 46(4), pages 866-906, August.
    10. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    11. Vieira, Elizabeth S. & Lepori, Benedetto, 2016. "The growth process of higher education institutions and public policies," Journal of Informetrics, Elsevier, vol. 10(1), pages 286-298.
    12. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    13. Daniele Checchi & Irene Mazzotta & Sandro Momigliano & Francesco Olivanti, 2020. "Convergence or polarisation? The impact of research assessment exercises in the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1439-1455, August.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2016. "The north–south divide in the Italian higher education system," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2093-2117, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    2. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    3. Fedderke, J.W. & Goldschmidt, M., 2015. "Does massive funding support of researchers work?: Evaluating the impact of the South African research chair funding initiative," Research Policy, Elsevier, vol. 44(2), pages 467-482.
    4. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: a comparison of peer review and bibliometrics rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 929-941, December.
    6. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    7. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of researchers’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 192-201.
    8. Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
    9. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Evaluating university research: Same performance indicator, different rankings," Journal of Informetrics, Elsevier, vol. 9(3), pages 514-525.
    10. Giovanni Abramo & Ciriaco Andrea D’Angelo & Myroslava Hladchenko, 2023. "Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4589-4609, August.
    11. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    12. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2014. "Are the authors of highly cited articles also the most productive ones?," Journal of Informetrics, Elsevier, vol. 8(1), pages 89-97.
    13. Shibayama, Sotaro & Baba, Yasunori, 2015. "Impact-oriented science policies and scientific publication practices: The case of life sciences in Japan," Research Policy, Elsevier, vol. 44(4), pages 936-950.
    14. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    15. Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    16. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    17. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    18. Li, Jiang & Sanderson, Mark & Willett, Peter & Norris, Michael & Oppenheim, Charles, 2010. "Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments," Journal of Informetrics, Elsevier, vol. 4(4), pages 554-563.
    19. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: the effects of changing the rules of the game during the game," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(1), pages 229-238, July.
    20. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jinfst:v:66:y:2015:i:11:p:2202-2214. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.