Variability and negligence: grant peer review panels evaluating impact ex ante

Building on extensive observations of grant review panels and interviews with panellists in five funding organizations, we explore how such panels assess societal impact. We find that impact assessments are accorded little weight and are prone to substantial reviewer variability unless competitions feature dedicated societal impact criteria with clearly assigned weight, which are scored collectively based on highly structured discussions. These findings cast new light on the general phenomenon of variability in grant assessments. While earlier literature has tended to see inter-rater variability as a flaw in the review process that should be rectified, an unavoidable part of the process, or a desirable feature, we advocate a more nuanced understanding. Introducing the concepts of productive and unproductive variability, we show the dual nature of variability and provide concrete policy recommendations for how unproductive variability, caused by differing interpretations or gaming of the organizational framework, can be reduced.