Risk and Reward in Science Funding: A Field Experiment on Grant Evaluation

Public money for research is distributed through peer-review but it has been criticized to disfavour risky science. Traditional review typically asks reviewers to compress multiple dimensions such as novelty, feasibility, impact, etc. into a single overall score, and this compression may penalise proposals whose potential is high but uncertain. The subjective expected utility (SEU) framework instead asks them to assess separately the utility and likelihood of a proposal's potential outcomes without issuing an overall compressed score. This paper examines whether structuring peer review in this way, systematically alters the ranking of grant proposals. Using a field experiment embedded in a real funding call, we have the same proposals evaluated under either traditional criteria or SEU criteria, with a treatment assignment that we argue to be as good as random. We find that the two methods produce very different orderings of the same applications: their percentile ranks are essentially uncorrelated, and individual applications shift considerably in relative position. SEU reorders applications along dimensions traditional review leaves implicit: 'High-Likelihood High-Utility' applications are ranked higher and 'Low-Likelihood Low-Utility' applications lower under SEU as compared to traditional review. Importantly, we do not observe a significant up- ranking of ‘Low-Likelihood High-Utility’ applications under SEU relative to the traditional review. On the contrary, 'High-Likelihood Low-Utility' application show a significant yet modest down-ranking under SEU relative to traditional review. These ranking differences translate into substantially different funding outcomes. Applying the funding cutoff that corresponds to the number of applications actually funded to each ranking, the 3 applications that would be funded under SEU were, without exception, those that referees had classified as high-likelihood (23 'High-Likelihood High-Utility' and 2 'High-Likelihood Low-Utility'). Traditional review, while also favouring high-likelihood submissions, would fund a mix across all four likelihood-utility categories, including the low-likelihood applications (roughly a third of the pool) that fail to clear the cutoff under SEU. On average, only 3.5 of the 25 applications funded under traditional review would also be funded under SEU, corresponding to a turnover rate of 86%. Our findings show that the structure of evaluation systematically shapes which applications rise to the top and, under a fixed funding cutoff, which would be funded, depending on whether referees issue a single overall judgment or score utility and likelihood separately.