Do “Evidence‐Based Policy” Clearinghouses Provide Good Advice for Local Policymakers?

Policymakers are often urged to rely on “evidence‐based policy” (EBP)—adopting only interventions proven effective (i.e., positive and statistically significant in multisite impact evaluations). EBP clearinghouses chronicle and rate tests of social policy interventions. But EBP clearinghouse standards are based almost entirely on internal validity. They largely ignore whether research findings from multisite trials apply to individual localities, where much of social policy is formulated. We develop a Bayesian model of the probability that the EBP rule is sound advice to local policymakers. The model allows a direct test of the probability of a correct policy decision under the EBP rule, its positive predictive value (PPV), and its negative predictive value (NPV)—the probabilities that an intervention deemed effective by that rule will in fact be effective in a particular site (PPV), and that an intervention deemed ineffective will not be effective in a particular site (NPV), given the true impact of the intervention. These intuitive, easily calculated probabilities are major contributions of this paper. In our illustrative analysis of six multisite randomized trials, we find that under the EBP clearinghouse rule the probability of a correct policy decision, PPV, and NPV are all unacceptably low unless the cross‐site impact heterogeneity is quite low.