IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v4y2010i3p211-220.html
   My bibliography  Save this article

A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications

Author

Listed:
  • Bornmann, Lutz
  • Leydesdorff, Loet
  • Van den Besselaar, Peter

Abstract

Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life sciences. In both fields, awarded applicants perform on average better than all rejected applicants. If only the most preeminent rejected applicants are considered in both fields, they score better than the awardees on citation impact. With regard to productivity we find differences between the fields. While the awardees in life sciences outperform on average the most preeminent rejected applicants, the situation is reversed in social sciences.

Suggested Citation

  • Bornmann, Lutz & Leydesdorff, Loet & Van den Besselaar, Peter, 2010. "A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications," Journal of Informetrics, Elsevier, vol. 4(3), pages 211-220.
    • Handle: RePEc:eee:infome:v:4:y:2010:i:3:p:211-220
    DOI: 10.1016/j.joi.2009.10.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157709000789
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2009.10.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Lutz Bornmann & Rüdiger Mutz & Hans‐Dieter Daniel, 2008. "Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 830-837, March.
    2. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    3. Hausman, Jerry & Hall, Bronwyn H & Griliches, Zvi, 1984. "Econometric Models for Count Data with an Application to the Patents-R&D Relationship," Econometrica, Econometric Society, vol. 52(4), pages 909-938, July.
    4. Lutz Bornmann & Hans‐Dieter Daniel, 2007. "Multiple publication on a single research study: Does it pay? The influence of number of research articles on total citation counts in biomedicine," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(8), pages 1100-1107, June.
    5. Lutz Bornmann & Gerlind Wallon & Anna Ledin, 2008. "Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data," Research Evaluation, Oxford University Press, vol. 17(2), pages 149-156, June.
    6. Lutz Bornmann & Gerlind Wallon & Anna Ledin, 2008. "Does the Committee Peer Review Select the Best Applicants for Funding? An Investigation of the Selection Process for Two European Molecular Biology Organization Programmes," PLOS ONE, Public Library of Science, vol. 3(10), pages 1-11, October.
    7. Moed, H. F. & Burger, W. J. M. & Frankfort, J. G. & Van Raan, A. F. J., 1985. "The use of bibliometric data for the measurement of university research performance," Research Policy, Elsevier, vol. 14(3), pages 131-149, June.
    8. J. Scott Long & Jeremy Freese, 2006. "Regression Models for Categorical Dependent Variables using Stata, 2nd Edition," Stata Press books, StataCorp LLC, edition 2, number long2, March.
    9. Bornmann, Lutz & Daniel, Hans-Dieter, 2007. "Convergent validation of peer review decisions using the h index," Journal of Informetrics, Elsevier, vol. 1(3), pages 204-213.
    10. Göran Melin & Rickard Danell, 2006. "The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden," Science and Public Policy, Oxford University Press, vol. 33(10), pages 702-712, December.
    11. Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kevin W. Boyack & Caleb Smith & Richard Klavans, 2018. "Toward predicting research proposal success," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 449-461, February.
    2. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    3. Bornmann, Lutz & Mutz, Rüdiger & Daniel, Hans-Dieter, 2010. "The h index research output measurement: Two approaches to enhance its accuracy," Journal of Informetrics, Elsevier, vol. 4(3), pages 407-414.
    4. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182.
    5. Benda, Wim G.G. & Engels, Tim C.E., 2011. "The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science," International Journal of Forecasting, Elsevier, vol. 27(1), pages 166-182, January.
    6. Bornmann, Lutz & Leydesdorff, Loet, 2013. "The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000," Journal of Informetrics, Elsevier, vol. 7(2), pages 286-291.
    7. van Raan, A. F. J. & van Leeuwen, Th. N., 2002. "Assessment of the scientific basis of interdisciplinary, applied research: Application of bibliometric methods in Nutrition and Food Research," Research Policy, Elsevier, vol. 31(4), pages 611-632, May.
    8. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    9. Sabrina Petersohn & Thomas Heinze, 2018. "Professionalization of bibliometric research assessment. Insights from the history of the Leiden Centre for Science and Technology Studies (CWTS)," Science and Public Policy, Oxford University Press, vol. 45(4), pages 565-578.
    10. Brady Lund, 2019. "Examination of correlates of H-index as a measure of research productivity for library and information science faculty in the United States and Canada," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 897-915, August.
    11. Bornmann, Lutz & Daniel, Hans-Dieter, 2009. "Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition," Journal of Informetrics, Elsevier, vol. 3(4), pages 348-352.
    12. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    13. Martin, Ben R., 2012. "The evolution of science policy and innovation studies," Research Policy, Elsevier, vol. 41(7), pages 1219-1239.
    14. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).
    15. Diekhof, Josefine & Cantner, Uwe, 2017. "Incumbents' responses to innovative entrants: A multi-country dynamic analysis," ZEW Discussion Papers 17-052, ZEW - Leibniz Centre for European Economic Research.
    16. Geuna, Aldo, 1998. "Determinants of university participation in EU-funded R & D cooperative projects," Research Policy, Elsevier, vol. 26(6), pages 677-687, February.
    17. Zhihui Zhang & Ying Cheng & Nian Cai Liu, 2015. "Improving the normalization effect of mean-based method from the perspective of optimization: optimization-based linear methods and their performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 587-607, January.
    18. Markus Eberhardt & Christian Helmers & Zhihong Yu, 2011. "Is the Dragon Learning to Fly? An Analysis of the Chinese Patent Explosion," Discussion Papers 11/16, University of Nottingham, GEP.
    19. Hird, Mackenzie D. & Pfotenhauer, Sebastian M., 2017. "How complex international partnerships shape domestic research clusters: Difference-in-difference network formation and research re-orientation in the MIT Portugal Program," Research Policy, Elsevier, vol. 46(3), pages 557-572.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:4:y:2010:i:3:p:211-220. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.