IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v18y2009i4p273-288.html
   My bibliography  Save this article

Past performance, peer review and project selection: a case study in the social and behavioral sciences

Author

Listed:
  • Peter van den Besselaar
  • Loet Leydesdorff

Abstract

Does past performance influence success in grant applications? We tested whether the decisions of the Netherlands Research Council for the Economic and Social Sciences correlate with the past performances of applicants in publications and citations, and with the results of the Council's peer reviews. The Council proves successful in distinguishing grant applicants with above-average from below-average performance, but within the former group there was no correlation between past performance and receiving a grant. When comparing the best-performing researchers who were denied funding with those who received it, the rejected researchers significantly outperformed the funded ones. The best rejected proposals score on average as high on the outcomes of the peer-review process as the accepted proposals. The Council successfully corrected for gender effects during the selection process. We explain why these findings may apply beyond this case. However, if research councils are not able to select the ‘best’ researchers, perhaps they should reconsider their mission. We discuss the role of research councils in the science system in terms of variation, innovation and quality control. Copyright , Beech Tree Publishing.

Suggested Citation

  • Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
    • Handle: RePEc:oup:rseval:v:18:y:2009:i:4:p:273-288
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.3152/095820209X475360
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pleun Arensbergen & Inge van der Weijden & Peter Besselaar, 2012. "Gender differences in scientific productivity: a persisting phenomenon?," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 857-868, December.
    2. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182.
    3. Peter van den Besselaar & Ulf Sandström, 2016. "Gender differences in research performance and its impact on careers: a longitudinal case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 143-162, January.
    4. Lepori, Benedetto, 2011. "Coordination modes in public funding systems," Research Policy, Elsevier, vol. 40(3), pages 355-367, April.
    5. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    6. Thelwall, Mike & Sud, Pardeep, 2016. "National, disciplinary and temporal variations in the extent to which articles with more authors have more impact: Evidence from a geometric field normalised citation indicator," Journal of Informetrics, Elsevier, vol. 10(1), pages 48-61.
    7. Kevin W. Boyack & Caleb Smith & Richard Klavans, 2018. "Toward predicting research proposal success," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 449-461, February.
    8. Marianne Hörlesberger & Ivana Roche & Dominique Besagni & Thomas Scherngell & Claire François & Pascal Cuxac & Edgar Schiebel & Michel Zitt & Dirk Holste, 2013. "A concept for inferring ‘frontier research’ in grant proposals," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 129-148, November.
    9. van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
    10. Paul Siu Fai Yip & Yunyu Xiao & Clifford Long Hin Wong & Terry Kit Fong Au, 2020. "Is there gender bias in research grant success in social sciences?: Hong Kong as a case study," Palgrave Communications, Palgrave Macmillan, vol. 7(1), pages 1-10, December.
    11. Benedetto Lepori & Valerio Veglio & Barbara Heller-Schuh & Thomas Scherngell & Michael Barber, 2015. "Participations to European Framework Programs of higher education institutions and their association with organizational characteristics," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2149-2178, December.
    12. Tobias Opthof & Loet Leydesdorff, 2011. "A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1011-1016, September.
    13. Bornmann, Lutz & Leydesdorff, Loet & Van den Besselaar, Peter, 2010. "A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications," Journal of Informetrics, Elsevier, vol. 4(3), pages 211-220.
    14. Vieira, Elizabeth S. & Lepori, Benedetto, 2016. "The growth process of higher education institutions and public policies," Journal of Informetrics, Elsevier, vol. 10(1), pages 286-298.
    15. Peter van den Besselaar & Ulf Sandström & Hélène Schiffbaenker, 2018. "Studying grant decision-making: a linguistic analysis of review reports," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 313-329, October.
    16. Sandström, Ulf & Van den Besselaar, Peter, 2018. "Funding, evaluation, and the performance of national research systems," Journal of Informetrics, Elsevier, vol. 12(1), pages 365-384.
    17. van den Besselaar, Peter, 2012. "Selection committee membership: Service or self-service," Journal of Informetrics, Elsevier, vol. 6(4), pages 580-585.
    18. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182, January.
    19. A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
    20. Thomas Zacharewicz & Noemi Pulido Pavón & Luis Antonio & Benedetto Lepori, 2023. "Do funding modes matter? A multilevel analysis of funding allocation mechanisms on university research performance," Research Evaluation, Oxford University Press, vol. 32(3), pages 545-556.
    21. Rüdiger Mutz & Lutz Bornmann & Hans-Dieter Daniel, 2015. "Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austr," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2321-2339, November.
    22. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    23. Jun-Ying Fu & Xu Zhang & Yun-Hua Zhao & He-Feng Tong & Dar-Zen Chen & Mu-Hsuan Huang, 2012. "Scientific production and citation impact: a bibliometric analysis in acupuncture over three decades," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 1061-1079, December.
    24. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    25. Irwin Feller, 2013. "Peer review and expert panels as techniques for evaluating the quality of academic research," Chapters, in: Albert N. Link & Nicholas S. Vonortas (ed.), Handbook on the Theory and Practice of Program Evaluation, chapter 5, pages 115-142, Edward Elgar Publishing.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:18:y:2009:i:4:p:273-288. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.