Field Experiments in the Science of Science: Lessons from Peer Review and the Evaluation of New Knowledge

Scientific evaluation and peer review govern the allocation of resources and the certification of knowledge in science, yet have rarely been studied causally. This chapter is an analytic synthesis of randomized experiments, re-evaluation studies, controlled comparisons and audit studies of scientific evaluation, most conducted within live evaluation systems at journals, conferences, and funding agencies and others in evaluations reconstructed for research purposes, published through 2025. I reconstruct each design in detail where the record allows, and representative designs where studies cluster, to establish what each experiment can and cannot identify. I develop a Q–A–R–S framework for analyzing the studies’ designs and results in common terms. It separates submission quality, author attributes, reviewer expertise, and the evaluation system, and foregrounds a basic constraint: new knowledge must be judged with an existing and incomplete stock of knowledge. Read together, the experiments suggest three claims of different standing: the first describes what the record establishes, the second rests on experimentally identified effects, and the third is a comparative hypothesis assembled across designs. First, system interventions often change participant behavior, and sometimes scores and decisions; the experiments establish these responses more clearly than any improvement in evaluative judgment. Second, who reviews, and what they know relative to the submission, systematically shapes evaluations of the same work. Third, the evidence is consistent with peer review performing more dependably as a filter of weak submissions than as a ranker of strong ones. The record leaves much of the design space unexplored, especially how individual evaluations are combined into decisions. I set out an agenda for a more cumulative experimental science of evaluation, including the use of artificial intelligence to experiment with evaluation design itself.