IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v25y2016i2p161-169..html
   My bibliography  Save this article

Evaluation practices and effects of indicator use—a literature review

Author

Listed:
  • Sarah de Rijcke
  • Paul F. Wouters
  • Alex D. Rushforth
  • Thomas P. Franssen
  • Björn Hammarfelt

Abstract

This review of the international literature on evaluation systems, evaluation practices, and metrics (mis)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014–5). The literature on evaluation systems, practices, and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our review presents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, ‘gaming’ of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-alone—after authorization by HEFCE.

Suggested Citation

  • Sarah de Rijcke & Paul F. Wouters & Alex D. Rushforth & Thomas P. Franssen & Björn Hammarfelt, 2016. "Evaluation practices and effects of indicator use—a literature review," Research Evaluation, Oxford University Press, vol. 25(2), pages 161-169.
    • Handle: RePEc:oup:rseval:v:25:y:2016:i:2:p:161-169.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvv038
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kaare Aagaard & Carter Bloch & Jesper W. Schneider, 2015. "Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator," Research Evaluation, Oxford University Press, vol. 24(2), pages 106-117.
    2. Harley, Diane & Acord, Sophia Krzys & Earl-Novell, Sarah & Lawrence, Shannon & King, C. Judson, 2010. "Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines - Executive Summary," University of California at Berkeley, Center for Studies in Higher Education qt0kr8s78v, Center for Studies in Higher Education, UC Berkeley.
    3. Kaare Aagaard, 2015. "How incentives trickle down: Local use of a national bibliometric indicator system," Science and Public Policy, Oxford University Press, vol. 42(5), pages 725-737.
    4. Harley, Diane & Acord, Sophia Krzys & Earl-Novell, Sarah & Lawrence, Shannon & King, C. Judson, 2010. "Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines," University of California at Berkeley, Center for Studies in Higher Education qt15x7385g, Center for Studies in Higher Education, UC Berkeley.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chieh Liu & Mu-Hsuan Huang, 2022. "Exploring the relationships between altmetric counts and citations of papers in different academic fields based on co-occurrence analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4939-4958, August.
    2. Alessio J. G. Brown & Klaus F. Zimmermann, 2017. "Three decades of publishing research in population economics," Journal of Population Economics, Springer;European Society for Population Economics, vol. 30(1), pages 11-27, January.
    3. Loliya Agbani Akobo, 2018. "Action learning through radio: exploring conceptual views and lived experiences of women entrepreneurs," Action Learning: Research and Practice, Taylor & Francis Journals, vol. 15(3), pages 235-248, September.
    4. Linda Sīle & Raf Vanderstraeten, 2019. "Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 71-91, January.
    5. Hamid R. Jamali & David Nicholas & Eti Herman, 2016. "Scholarly reputation in the digital age and the role of emerging platforms and mechanisms," Research Evaluation, Oxford University Press, vol. 25(1), pages 37-49.
    6. Kulczycki, Emanuel & Korzeń, Marcin & Korytkowski, Przemysław, 2017. "Toward an excellence-based research funding system: Evidence from Poland," Journal of Informetrics, Elsevier, vol. 11(1), pages 282-298.
    7. Renata Kudaibergenova & Sandugash Uzakbay & Asselya Makanova & Kymbat Ramadinkyzy & Erlan Kistaubayev & Ruslan Dussekeev & Kadyrzhan Smagulov, 2022. "Managing publication change at Al-Farabi Kazakh National University: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 453-479, January.
    8. Katarina Krapež, 2022. "Advancing Self-Evaluative and Self-Regulatory Mechanisms of Scholarly Journals: Editors’ Perspectives on What Needs to Be Improved in the Editorial Process," Publications, MDPI, vol. 10(1), pages 1-18, March.
    9. Fei Shu & Wei Quan & Bikun Chen & Junping Qiu & Cassidy R. Sugimoto & Vincent Larivière, 2020. "The role of Web of Science publications in China’s tenure system," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1683-1695, March.
    10. Colin Gallagher & Dean Lusher & Johan Koskinen & Bopha Roden & Peng Wang & Aaron Gosling & Anastasios Polyzos & Martina Stenzel & Sarah Hegarty & Thomas Spurling & Gregory Simpson, 2023. "Network patterns of university-industry collaboration: A case study of the chemical sciences in Australia," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4559-4588, August.
    11. Jingfeng Xia, 2013. "The Open Access Divide," Publications, MDPI, vol. 1(3), pages 1-27, October.
    12. Erin C McKiernan, 2017. "Imagining the “open” university: Sharing scholarship to improve research and education," PLOS Biology, Public Library of Science, vol. 15(10), pages 1-25, October.
    13. William E Savage & Anthony J Olejniczak, 2022. "More journal articles and fewer books: Publication practices in the social sciences in the 2010’s," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-16, February.
    14. Claudiu Vasile Kifor & Ana Maria Benedek & Ioan Sîrbu & Roxana Florența Săvescu, 2023. "Institutional drivers of research productivity: a canonical multivariate analysis of Romanian public universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2233-2258, April.
    15. Daniella Bayle Deutz & Thea Marie Drachen & Dorte Drongstrup & Niels Opstrup & Charlotte Wien, 2021. "Quantitative quality: a study on how performance-based measures may change the publication patterns of Danish researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3303-3320, April.
    16. Esteban Morales & Erin C McKiernan & Meredith T Niles & Lesley Schimanski & Juan Pablo Alperin, 2021. "How faculty define quality, prestige, and impact of academic journals," PLOS ONE, Public Library of Science, vol. 16(10), pages 1-13, October.
    17. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    18. William H. Walters & Susanne Markgren, 2019. "Do faculty journal selections correspond to objective indicators of citation impact? Results for 20 academic departments at Manhattan College," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 321-337, January.
    19. Laura Himanen & Hanna-Mari Puuska, 2022. "Does monitoring performance act as an incentive for improving research performance? National and organizational level analysis of Finnish universities," Research Evaluation, Oxford University Press, vol. 31(2), pages 236-248.
    20. Elea Giménez-Toledo & Jorge Mañana-Rodríguez & Tim C. E. Engels & Peter Ingwersen & Janne Pölönen & Gunnar Sivertsen & Frederik T. Verleysen & Alesia A. Zuccala, 2016. "Taking scholarly books into account: current developments in five European countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 685-699, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:25:y:2016:i:2:p:161-169.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.