Uncertainty in Compound Decisions

Researchers and policymakers frequently make a large number of decisions simultaneously over many units using noisy estimates of unit-level effects. Researchers often assume that unit-level effects are drawn from a common effect distribution, allowing for optimal decision-making. In applications, however, this effect distribution is unknown to the econometrician, introducing uncertainty in the ultimate decision-making problem. This paper considers the problem of quantifying this uncertainty. For instance, researchers may seek to understand how an oracle decision maker with knowledge of the effect distribution would compare different feasible decisions, or understand which decisions an oracle would plausibly make. Analyzing these questions from the perspective of an oracle decision maker involves comparing the conditional means of unit-level effects given estimates thereof, suggesting a sufficient inferential objective of uniform coverage of these conditional means. Based on an equivalence between uniform coverage of conditional means and multiple testing, I develop methods that satisfy this coverage. Finally, I consider an application to the problem of identifying high-economic-mobility neighborhoods. My methods sharpen inference relative to existing methods and identify units that can be confidently included in oracle selections.