2020 Rossi Award Lecture: The Evolving Art of Program Evaluation

Evaluation of public programs has undergone many changes over the past four decades since Peter Rossi coined his “Iron Law†of program evaluation: “The expected value of any net impact assessment of any large-scale social program is zero.†While that assessment may be somewhat overstated, the essence still holds. The failures far outnumber the successes, and the estimated favorable effects are rarely sizeable. Despite this grim assessment, much can be learned from “failed†experiments, and from ones that are successful in only some sites or subgroups. Advances in study design, statistical models, data, and how inferences are drawn from estimates have substantially improved our analyses and will continue to do so. However, the most actual learning about “what works†(and why, when, and where) is likely to come from gathering more detailed and comprehensive data on how the intervention was implemented and attempting to link that data to estimated impacts. Researchers need detailed data on the target population served, the content of the intervention, and the process by which it is delivered to participating service providers and individuals. Two examples presented here illustrate how researchers drew useful broader lessons from impact estimates for a set of related programs. Rossi posited three reasons most interventions fail—wrong question, wrong intervention, poor implementation. Speeding the accumulation of wisdom about how social programs can best help vulnerable populations will require that researchers work closely with program funders, developers, operators, and participants to gather and interpret these detailed data about program implementation.