Experimenting with Grant Peer Review: A Mixed Methods Case Study of the Effects on Time Use and the Quality of Reviewing

Rejection of review invitations due to time constraints is putting pressure on the peer review system, showing that less time-consuming ways of reviewing are needed. This paper presents the results of a field experiment with a new format for grant peer review and answers the question of whether this new format is less time-consuming while still providing high-quality reviews. In the new approach, the Peer Circle (PC), a team of reviewers collectively evaluates several grant applications. The PC was applied to four fields and compared with four similar fields using conventional peer review. Qualitative and quantitative methods have been used to analyze heterogeneous data such as interviews with and a survey of the peer reviewers; text analysis of the review reports; and statistical analysis of bibliometric applicant data. The comparison suggests that the PC saves time and enlarges the reviewer population considerably. Most reviewers felt that the quality of the PC evaluations was at least as good as that of the conventional evaluations, if not better. Given these findings, the experiment is now continued on a much larger scale. Apart from that, the theoretical implication is that the way of organizing peer review has an important effect on the functioning of the system.