IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i8d10.1007_s11192-022-04469-5.html
   My bibliography  Save this article

Reasons and consequences of changes in Russian research assessment policies

Author

Listed:
  • Denis Kosyakov

    (Russian Research Institute of Economics, Politics and Law in Science and Technology)

  • Andrey Guskov

    (Russian Research Institute of Economics, Politics and Law in Science and Technology)

Abstract

The transition to quantitative methods of assessing scientific performance in Russian science in the early 2010s led to a sharp increase in publication activity. It has been accompanied by inflation—an outstripping increase in the publications number due to not always justified collaborations that allow results to be credited to several organisations and a decrease in the average quality of the publication flow. Growth has been achieved mainly through publications in conference proceedings and low-rated journals. Since 2020, the Ministry of Science and Higher Education of the Russian Federation has applied a new assessment methodology that implements the model of collaboration penalty and quartile bonuses: fractional count of publications and multiplying factors for ones in high-impact journals. The analysis carried out as part of this study shows that the primary beneficiaries of this transformation are academic institutions specializing in Chemistry, Materials Science, and Life Sciences. Organisations traditionally strong in Physics and Astronomy are somewhat inferior to the leaders, mainly due to active participation in mega-collaborations. A separate scale of quality factors gives unjustified advantages to social science organisations, which may exacerbate the number of scientific ethics breaches. In general, the new system should help mitigate distortions in the development of Russian science. However, some side effects of its implementation need increased attention and will probably require adjustments in the methodology.

Suggested Citation

  • Denis Kosyakov & Andrey Guskov, 2022. "Reasons and consequences of changes in Russian research assessment policies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4609-4630, August.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:8:d:10.1007_s11192-022-04469-5
    DOI: 10.1007/s11192-022-04469-5
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04469-5
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04469-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. John P. A. Ioannidis & Richard Klavans & Kevin W. Boyack, 2018. "Thousands of scientists publish a paper every five days," Nature, Nature, vol. 561(7722), pages 167-169, September.
    2. Ludo Waltman & Clara Calero‐Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan van Eck & Thed N. van Leeuwen & Anthony F.J. van Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    3. Alison Abbott & David Cyranoski & Nicola Jones & Brendan Maher & Quirin Schiermeier & Richard Van Noorden, 2010. "Metrics: Do metrics matter?," Nature, Nature, vol. 465(7300), pages 860-862, June.
    4. Lin, Chi-Shiou & Huang, Mu-Hsuan & Chen, Dar-Zen, 2013. "The influences of counting methods on university rankings based on paper count and citation count," Journal of Informetrics, Elsevier, vol. 7(3), pages 611-621.
    5. Tatiana Marina & Ivan Sterligov, 2021. "Prevalence of potentially predatory publishing in Scopus on the country level," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5019-5077, June.
    6. Hanna Hottenrott & Michael E. Rose & Cornelia Lawson, 2021. "The rise of multiple institutional affiliations in academia," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(8), pages 1039-1058, August.
    7. Matveeva, Nataliya & Sterligov, Ivan & Yudkevich, Maria, 2021. "The effect of Russian University Excellence Initiative on publications and collaboration patterns," Journal of Informetrics, Elsevier, vol. 15(1).
    8. Henk F. Moed & Valentina Markusova & Mark Akoev, 2018. "Trends in Russian research output indexed in Scopus and Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 1153-1180, August.
    9. Vincent Larivière & Rodrigo Costas, 2016. "How Many Is Too Many? On the Relationship between Research Productivity and Impact," PLOS ONE, Public Library of Science, vol. 11(9), pages 1-10, September.
    10. Abramo, Giovanni & D’Angelo, Andrea Ciriaco & Murgia, Gianluca, 2017. "The relationship among research productivity, research collaboration, and their determinants," Journal of Informetrics, Elsevier, vol. 11(4), pages 1016-1030.
    11. Tatiana Marina & Ivan Sterligov, 2021. "Correction to: Prevalence of potentially predatory publishing in Scopus on the country level," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5079-5079, June.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2013. "Measuring institutional research productivity for the life sciences: the importance of accounting for the order of authors in the byline," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 779-795, December.
    13. Mathias Binswanger, 2015. "How Nonsense Became Excellence: Forcing Professors to Publish," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 19-32, Springer.
    14. Svetlana Parfenova & Ekaterina Grishakina & Viktor Bogatov & Dmitry Zolotarev, 2017. "Publication Landscape of the Russian Science," Science Governance and Scientometrics Journal, Russian Research Institute of Economics, Politics and Law in Science and Technology (RIEPL), vol. 12(1), pages 53-79, March.
    15. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Solazzi, Marco, 2011. "Are researchers that collaborate more at the international level top performers? An investigation on the Italian university system," Journal of Informetrics, Elsevier, vol. 5(1), pages 204-213.
    16. Sivertsen, Gunnar & Rousseau, Ronald & Zhang, Lin, 2019. "Measuring scientific contributions with modified fractional counting," Journal of Informetrics, Elsevier, vol. 13(2), pages 679-694.
    17. Ludo Waltman & Clara Calero-Medina & Joost Kosten & Ed C.M. Noyons & Robert J.W. Tijssen & Nees Jan Eck & Thed N. Leeuwen & Anthony F.J. Raan & Martijn S. Visser & Paul Wouters, 2012. "The Leiden ranking 2011/2012: Data collection, indicators, and interpretation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2419-2432, December.
    18. Leo Egghe & Ronald Rousseau & Guido Van Hooydonk, 2000. "Methods for accrediting publications to authors or countries: Consequences for evaluation studies," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(2), pages 145-157.
    19. Andrey Guskov & Denis Kosyakov & Irina Selivanova, 2016. "Scientometric research in Russia: impact of science policy changes," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(1), pages 287-303, April.
    20. Mu-Hsuan Huang & Chi-Shiou Lin & Dar-Zen Chen, 2011. "Counting methods, country rank changes, and counting inflation in the assessment of national research productivity and impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(12), pages 2427-2436, December.
    21. Henk F. Moed & Gali Halevi, 2015. "Multidimensional assessment of scholarly research impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 1988-2002, October.
    22. Daniele Fanelli, 2010. "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data," PLOS ONE, Public Library of Science, vol. 5(4), pages 1-7, April.
    23. Mu‐Hsuan Huang & Chi‐Shiou Lin & Dar‐Zen Chen, 2011. "Counting methods, country rank changes, and counting inflation in the assessment of national research productivity and impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(12), pages 2427-2436, December.
    24. Edmilson J. T. Manganote & Peter A. Schulz & Carlos Henrique Brito Cruz, 2016. "Effect of high energy physics large collaborations on higher education institutions citations and rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 813-826, November.
    25. Andrey E. Guskov & Denis V. Kosyakov & Irina V. Selivanova, 2018. "Boosting research productivity in top Russian universities: the circumstances of breakthrough," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 1053-1080, November.
    26. Sandro Tarkhan-Mouravi, 2020. "Traditional indicators inflate some countries’ scientific impact over 10 times," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 337-356, April.
    27. Perianes-Rodriguez, Antonio & Waltman, Ludo & van Eck, Nees Jan, 2016. "Constructing bibliometric networks: A comparison between full and fractional counting," Journal of Informetrics, Elsevier, vol. 10(4), pages 1178-1195.
    28. Waltman, Ludo & van Eck, Nees Jan, 2015. "Field-normalized citation impact indicators and the choice of an appropriate counting method," Journal of Informetrics, Elsevier, vol. 9(4), pages 872-894.
    29. Maia Chankseliani & Andrey Lovakov & Vladimir Pislyakov, 2021. "A big picture: bibliometric study of academic publications from post-Soviet countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8701-8730, October.
    30. Helmut A. Abt, 2007. "The frequencies of multinational papers in various sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(1), pages 105-115, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Jeffrey Demaine, 2022. "Fractionalization of research impact reveals global trends in university collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2235-2247, May.
    3. Nataliya Matveeva & Ivan Sterligov & Andrey Lovakov, 2022. "International scientific collaboration of post-Soviet countries: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1583-1607, March.
    4. Korytkowski, Przemyslaw & Kulczycki, Emanuel, 2019. "Publication counting methods for a national research evaluation exercise," Journal of Informetrics, Elsevier, vol. 13(3), pages 804-816.
    5. Rahman, Mohammad Tariqur & Regenstein, Joe Mac & Kassim, Noor Lide Abu & Haque, Nazmul, 2017. "The need to quantify authors’ relative intellectual contributions in a multi-author paper," Journal of Informetrics, Elsevier, vol. 11(1), pages 275-281.
    6. Yu, Xiaoyao & Szymanski, Boleslaw K. & Jia, Tao, 2021. "Become a better you: Correlation between the change of research direction and the change of scientific performance," Journal of Informetrics, Elsevier, vol. 15(3).
    7. Fairclough, Ruth & Thelwall, Mike, 2015. "National research impact indicators from Mendeley readers," Journal of Informetrics, Elsevier, vol. 9(4), pages 845-859.
    8. Sandro Tarkhan-Mouravi, 2020. "Traditional indicators inflate some countries’ scientific impact over 10 times," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 337-356, April.
    9. Waltman, Ludo & van Eck, Nees Jan, 2015. "Field-normalized citation impact indicators and the choice of an appropriate counting method," Journal of Informetrics, Elsevier, vol. 9(4), pages 872-894.
    10. Pär Sundling, 2023. "Author contributions and allocation of authorship credit: testing the validity of different counting methods in the field of chemical biology," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 2737-2762, May.
    11. Csomós, György, 2020. "Introducing recalibrated academic performance indicators in the evaluation of individuals’ research performance: A case study from Eastern Europe," Journal of Informetrics, Elsevier, vol. 14(4).
    12. Potter, Ross W.K. & Szomszor, Martin & Adams, Jonathan, 2020. "Interpreting CNCIs on a country-scale: The effect of domestic and international collaboration type," Journal of Informetrics, Elsevier, vol. 14(4).
    13. Cinzia Daraio & Simone Di Leo & Loet Leydesdorff, 2022. "Using the Leiden Rankings as a Heuristics: Evidence from Italian universities in the European landscape," LEM Papers Series 2022/08, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    14. Leporia, Benedetto & Geuna, Aldo & Mira, Antonietta, 2018. "Scientific Output of US and European Universities Scales Super-linearly with Resources," Department of Economics and Statistics Cognetti de Martiis LEI & BRICK - Laboratory of Economics of Innovation "Franco Momigliano", Bureau of Research in Innovation, Complexity and Knowledge, Collegio 201806, University of Turin.
    15. Félix Moya-Anegón & Vicente P. Guerrero-Bote & Lutz Bornmann & Henk F. Moed, 2013. "The research guarantors of scientific papers and the output counting: a promising new approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 421-434, November.
    16. Fabio S. V. Silva & Peter A. Schulz & Everard C. M. Noyons, 2019. "Co-authorship networks and research impact in large research facilities: benchmarking internal reports and bibliometric databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 93-108, January.
    17. Timur Gareev & Irina Peker, 2023. "Quantity versus quality in publication activity: knowledge production at the regional level," Papers 2311.08830, arXiv.org.
    18. Jesper W. Schneider & Thed Leeuwen & Martijn Visser & Kaare Aagaard, 2019. "Examining national citation impact by comparing developments in a fixed and a dynamic journal set," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 973-985, May.
    19. Ross W. K. Potter & Martin Szomszor & Jonathan Adams, 2022. "Comparing standard, collaboration and fractional CNCI at the institutional level: Consequences for performance evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7435-7448, December.
    20. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:8:d:10.1007_s11192-022-04469-5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.