How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework

There is growing interest in the wider impact of research outside academia. In particular, there is concern about the extent to which the topic and principal beneficiaries of such impact can be foreseen before research is even funded. Does research impact tend to emerge largely as planned, or are eventual impacts unrecognisable from initial plans? To explore the unexpectedness of impact, we draw on one of the largest research impact assessment exercises in the world - the UK’s Research Excellence Framework. We exploit REF impact traced back to research funding applications, as a dataset of 2,194 case-grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex-ante and ex-post impact. We found impact claims in these case-grant pairs are often congruent with each other, with 76% showing alignment. This indicates that, conditional on REF-inclusion, research delivers impact with largely the same orientation as envisaged at the outset of their projects. This is especially so when research is co-produced with focal stakeholders. These results offer a distinct complement to other preliminary studies on the REF, which have tended to emphasise the vast diversity and complexity of research impact. Our findings suggest that research impact is far from random, and that there is wide scope for policy intervention. In particular, we highlight the potentially important role that collaboration plays in setting the direction of research impact. However, the findings may also signal that the REF is preferentially excluding unpredicted impacts and indirect impacts. This selection effect, where a qualitatively different class of impacts are crowded out, may well be exacerbated as the share of REF rewards is weighted further towards the impact criterion.