IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v85y2010i2d10.1007_s11192-010-0230-8.html
   My bibliography  Save this article

Scientometric indicators: peer-review, bibliometric methods and conflict of interests

Author

Listed:
  • Primož Južnič

    (University of Ljubljana)

  • Stojan Pečlin

    (Slovenian Research Agency (ARRS))

  • Matjaž Žaucer

    (Central Technological Library at the University of Ljubljana)

  • Tilen Mandelj

    (Central Technological Library at the University of Ljubljana)

  • Miro Pušnik

    (Central Technological Library at the University of Ljubljana)

  • Franci Demšar

    (Slovenian Research Agency (ARRS))

Abstract

The paper discusses the role of scientometric indicators in peer-review selection of research project proposals. An ex post facto evaluation was made of three calls for research project proposals in Slovenia: 2003 with a peer review system designed in a way that conflict of interest was not avoided effectively, 2005 with a sound international peer-review system with minimized conflict of interest influence but a limited number of reviewers, and 2008 with a combination of scientometric indicators and a sound international peer review with minimized conflict of interest influence. The hypothesis was that the three different peer review systems would have different correlations with the same set of scientometric indicators. In the last two decision-making systems (2005 and 2008) where conflict of interest was effectively avoided, we have a high percentage (65%) of projects that would have been selected in the call irrespective of the method (peer review or bibliometrics solely). In contrast, in the 2003 call there is a significantly smaller percentage (49%) of projects that would have been selected in the call irrespective of the method (peer review or bibliometrics solely). It was shown that while scientometric indicators can hardly replace the peer-review system as the ultimate decision-making and support system, they can reveal its weaknesses on one hand and on the other can verify peer-review scores and minimize conflict of interest if necessary.

Suggested Citation

  • Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    • Handle: RePEc:spr:scient:v:85:y:2010:i:2:d:10.1007_s11192-010-0230-8
    DOI: 10.1007/s11192-010-0230-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-010-0230-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-010-0230-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Thed N. Van Leeuwen & Martijn S. Visser & Henk F. Moed & Ton J. Nederhof & Anthony F. J. Van Raan, 2003. "The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 57(2), pages 257-280, June.
    2. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    3. H. F. Moed & Th. N. Leeuwen & J. Reedijk, 1999. "Towards appropriate indicators of journal impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 575-589, November.
    4. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    5. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 2001. "Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research," Research Policy, Elsevier, vol. 30(3), pages 357-361, March.
    6. Lutz Bornmann & Hans-Dieter Daniel, 2006. "Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 427-440, September.
    7. Peter Weingart, 2005. "Impact of bibliometrics upon the science system: Inadvertent consequences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 117-131, January.
    8. Martin Reinhart, 2009. "Peer review of grant applications in biology and medicine. Reliability, fairness, and validity," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 789-809, December.
    9. Peder Olesen Larsen, 2008. "The state of the art in publication counting," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(2), pages 235-251, November.
    10. John Rigby, 2009. "Comparing the scientific quality achieved by funding instruments for single grant holders and for collaborative networks within a research system: Some observations," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(1), pages 145-164, January.
    11. Mario Coccia, 2009. "Research performance and bureaucracy within public research labs," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(1), pages 93-107, April.
    12. Jerome K. Vanclay, 2009. "Bias in the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(1), pages 3-12, January.
    13. Wolfgang Glänzel & Koenraad Debackere & Bart Thijs & András Schubert, 2006. "A concise review on the role of author self-citations in information science, bibliometrics and science policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(2), pages 263-277, May.
    14. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Romina Rodela, 2016. "On the use of databases about research performance: comments on Karlovčec and Mladenić (2015) and others using the SICRIS database," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2151-2157, December.
    2. Wei-dong Zhu & Fang Liu & Yu-wang Chen & Jian-bo Yang & Dong-ling Xu & Dong-peng Wang, 2015. "Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 1469-1490, December.
    3. Dejan Pajić, 2015. "Globalization of the social sciences in Eastern Europe: genuine breakthrough or a slippery slope of the research evaluation practice?," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2131-2150, March.
    4. Teja Koler-Povh & Primož Južnič & Goran Turk, 2014. "Impact of open access on citation of scholarly publications in the field of civil engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1033-1045, February.
    5. Marianne Hörlesberger & Ivana Roche & Dominique Besagni & Thomas Scherngell & Claire François & Pascal Cuxac & Edgar Schiebel & Michel Zitt & Dirk Holste, 2013. "A concept for inferring ‘frontier research’ in grant proposals," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 129-148, November.
    6. Hamid Bouabid, 2014. "Science and technology metrics for research policy evaluation: some insights from a Moroccan experience," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 899-915, October.
    7. Fang Liu & Wei-dong Zhu & Yu-wang Chen & Dong-ling Xu & Jian-bo Yang, 2017. "Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1501-1519, June.
    8. Zhang, Baolong & Wang, Hao & Deng, Sanhong & Su, Xinning, 2020. "Measurement and analysis of Chinese journal discriminative capacity," Journal of Informetrics, Elsevier, vol. 14(1).
    9. Stojan Pečlin & Primož Južnič & Rok Blagus & Mojca Čižek Sajko & Janez Stare, 2012. "Effects of international collaboration and status of journal on impact of papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 937-948, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    2. Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
    3. Li, Jiang & Sanderson, Mark & Willett, Peter & Norris, Michael & Oppenheim, Charles, 2010. "Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments," Journal of Informetrics, Elsevier, vol. 4(4), pages 554-563.
    4. Giovanni Abramo & Ciriaco Andrea D'Angelo, 2015. "The VQR, Italy's second national research assessment: Methodological failures and ranking distortions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2202-2214, November.
    5. Emanuela Reale & Anna Barbara & Antonio Costantini, 2006. "Peer review for the evaluation of the academic research: the Italian experience," CERIS Working Paper 200615, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    6. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    7. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    8. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    9. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    10. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    11. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    12. Ulrich Schmoch & Torben Schubert, 2008. "Are international co-publications an indicator for quality of scientific research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(3), pages 361-377, March.
    13. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    15. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    16. Fedderke, J.W. & Goldschmidt, M., 2015. "Does massive funding support of researchers work?: Evaluating the impact of the South African research chair funding initiative," Research Policy, Elsevier, vol. 44(2), pages 467-482.
    17. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    18. Hui-Zhen Fu & Yuh-Shan Ho, 2013. "Comparison of independent research of China’s top universities using bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 259-276, July.
    19. Mike Thelwall, 2017. "Are Mendeley reader counts useful impact indicators in all fields?," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1721-1731, December.
    20. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:85:y:2010:i:2:d:10.1007_s11192-010-0230-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.