IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i2s1751157722000414.html
   My bibliography  Save this article

What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports

Author

Listed:
  • Hren, Darko
  • Pina, David G.
  • Norman, Christopher R.
  • Marušić, Ana

Abstract

The evaluation of grant proposals is an essential aspect of competitive research funding. Funding bodies and agencies rely in many instances on external peer reviewers for grant assessment. Most of the research available is about quantitative aspects of this assessment, and there is little evidence from qualitative studies. We used a combination of machine learning and qualitative analysis methods to analyse the reviewers' comments in evaluation reports from 3667 grant applications to the Initial Training Networks (ITN) of the Marie Curie Actions under the Seventh Framework Programme (FP7). Our results show that the reviewers' comments for each evaluation criterion were aligned with the Action's prespecified criteria and that the evaluation outcome was more influenced by the proposals’ weaknesses than by their strengths.

Suggested Citation

  • Hren, Darko & Pina, David G. & Norman, Christopher R. & Marušić, Ana, 2022. "What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports," Journal of Informetrics, Elsevier, vol. 16(2).
    • Handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000414
    DOI: 10.1016/j.joi.2022.101289
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000414
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2022.101289?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Győrffy, Balázs & Herman, Péter & Szabó, István, 2020. "Research funding: past performance is a stronger predictor of future scientific output than reviewer scores," Journal of Informetrics, Elsevier, vol. 14(3).
    2. Tohalino, Jorge A.V. & Amancio, Diego R., 2022. "On predicting research grants productivity via machine learning," Journal of Informetrics, Elsevier, vol. 16(2).
    3. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    4. David G Pina & Darko Hren & Ana Marušić, 2015. "Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    5. Marco Seeber & Jef Vlegels & Mattia Cattaneo, 2022. "Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(8), pages 1106-1126, August.
    6. Lutz Bornmann & Markus Wolf & Hans-Dieter Daniel, 2012. "Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 843-856, June.
    7. Dzieżyc, Maciej & Kazienko, Przemysław, 2022. "Effectiveness of research grants funded by European Research Council and Polish National Science Centre," Journal of Informetrics, Elsevier, vol. 16(1).
    8. Peter van den Besselaar & Ulf Sandström & Hélène Schiffbaenker, 2018. "Studying grant decision-making: a linguistic analysis of review reports," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 313-329, October.
    9. Bayindir, Esra Eren & Gurdal, Mehmet Yigit & Saglam, Ismail, 2019. "A Game Theoretic Approach to Peer Review of Grant Proposals," Journal of Informetrics, Elsevier, vol. 13(4).
    10. Ayoubi, Charles & Pezzoni, Michele & Visentin, Fabiana, 2019. "The important thing is not to win, it is to take part: What if scientists benefit from participating in research grant competitions?," Research Policy, Elsevier, vol. 48(1), pages 84-97.
    11. Ron Johnston & Kelvyn Jones & David Manley, 2018. "Confounding and collinearity in regression analysis: a cautionary tale and an alternative procedure, illustrated by studies of British voting behaviour," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(4), pages 1957-1976, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sven E. Hug, 2024. "How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(3), pages 1231-1253, March.
    2. Andrijana Perković Paloš & Antonija Mijatović & Ivan Buljan & Daniel Garcia-Costa & Elena Álvarez-García & Francisco Grimaldo & Ana Marušić, 2023. "Linguistic and semantic characteristics of articles and peer review reports in Social Sciences and Medical and Health Sciences: analysis of articles published in Open Research Central," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4707-4729, August.
    3. Gabriel Okasa & Alberto de Le'on & Michaela Strinzel & Anne Jorstad & Katrin Milzow & Matthias Egger & Stefan Muller, 2024. "A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports," Papers 2411.16662, arXiv.org, revised Dec 2024.
    4. Tóth, Tamás & Demeter, Márton & Csuhai, Sándor & Major, Zsolt Balázs, 2024. "When career-boosting is on the line: Equity and inequality in grant evaluation, productivity, and the educational backgrounds of Marie Skłodowska-Curie Actions individual fellows in social sciences an," Journal of Informetrics, Elsevier, vol. 18(2).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shan Jiang, 2021. "Understanding authors' psychological reactions to peer reviews: a text mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6085-6103, July.
    2. Seeber, Marco & Alon, Ilan & Pina, David G. & Piro, Fredrik Niclas & Seeber, Michele, 2022. "Predictors of applying for and winning an ERC Proof-of-Concept grant: An automated machine learning model," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    3. Kok, Holmer & Faems, Dries & de Faria, Pedro, 2022. "Pork Barrel or Barrel of Gold? Examining the performance implications of earmarking in public R&D grants," Research Policy, Elsevier, vol. 51(7).
    4. Giada Baldessarelli & Nathalie Lazaric & Michele Pezzoni, 2022. "Organizational routines: Evolution in the research landscape of two core communities," Journal of Evolutionary Economics, Springer, vol. 32(4), pages 1119-1154, September.
    5. Vivek Singh & Purnendu Sardar & Sukha Ranjan Samadder & Dheeraj Kumar & Vasant Govind K. Villuri, 2024. "Predicting the future landscape of Dhanbad District: an analysis of land-use change and urban sprawl through cloud computing and neural networks," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 26(12), pages 30749-30770, December.
    6. Kevin Escoz Barragan & Felix Simon Rudolf Becker, 2025. "Keeping pace with the digital transformation — exploring the digital orientation of SMEs," Small Business Economics, Springer, vol. 64(3), pages 1361-1385, March.
    7. Basheer Kalash & Sarah Guillou & Lionel Nesta & Michele Pezzoni, 2024. "Does Lab Funding Matter for the Technological Application of Scientific Research? An Empirical Analysis of French Labs," Annals of Economics and Statistics, GENES, issue 153, pages 39-76.
    8. Tim Heubeck & Annina Ahrens, 2025. "Governing the Responsible Investment of Slack Resources in Environmental, Social, and Governance (ESG) Performance: How Beneficial are CSR Committees?," Journal of Business Ethics, Springer, vol. 198(2), pages 365-385, May.
    9. Han-Sol Lee & Sergey U. Chernikov & Szabolcs Nagy & Ekaterina A. Degtereva, 2022. "The Impact of National Culture on Innovation: A Comparative Analysis between Developed and Developing Nations during the Pre- and Post-Crisis Period 2007–2021," Social Sciences, MDPI, vol. 11(11), pages 1-14, November.
    10. Samineh Sanatkar & Jenn Bartlett & Samuel Harvey & Isabelle Counson & David Lawrence, 2022. "The Influence of Stigma Perceptions on Employees’ Claims Experiences for Psychological Injuries: Re-Examination of a Cross-Sectional Survey among Australian Police and Emergency Service Personnel," IJERPH, MDPI, vol. 19(19), pages 1-16, September.
    11. Sven E. Hug, 2024. "How do referees integrate evaluation criteria into their overall judgment? Evidence from grant peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(3), pages 1231-1253, March.
    12. Giulio Giacomo Cantone, 2024. "How to measure interdisciplinary research? A systemic design for the model of measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4937-4982, August.
    13. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    14. Marton Demeter & Agnes Jele & Zsolt Balázs Major, 2022. "The model of maximum productivity for research universities SciVal author ranks, productivity, university rankings, and their implications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4335-4361, August.
    15. Tuyen Quang Tran & Huong Van Vu, 2021. "The impact of land fragmentation on food security in the North Central Coast, Vietnam," Asia and the Pacific Policy Studies, Wiley Blackwell, vol. 8(2), pages 327-345, May.
    16. Wang, Lili & Wang, Xianwen & Piro, Fredrik Niclas & Philipsen, Niels, 2019. "The effect of public funding on scientific performance: A comparison between China and the EU," MERIT Working Papers 2019-045, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    17. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    18. Mulugeta Mokria & Aster Gebrekirstos & Hadia Said & Kiros Hadgu & Niguse Hagazi & Workneh Dubale & Achim Bräuning, 2022. "Volume estimation models for avocado fruit," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-14, February.
    19. Joachim P. Hasebrook & Leonie Michalak & Anna Wessels & Sabine Koenig & Stefan Spierling & Stefan Kirmsse, 2022. "Green Behavior: Factors Influencing Behavioral Intention and Actual Environmental Behavior of Employees in the Financial Service Sector," Sustainability, MDPI, vol. 14(17), pages 1-35, August.
    20. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:2:s1751157722000414. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.