Flexible and Efficient Contextual Bandits with Heterogeneous Treatment Effect Oracles

Contextual bandit algorithms often estimate reward models to inform decision-making. However, true rewards can contain action- independent redundancies that are not relevant for decision-making. We show it is more data- efficient to estimate any function that explains the reward differences between actions, that is, the treatment effects. Motivated by this obser- vation, building on recent work on oracle-based bandit algorithms, we provide the first reduction of contextual bandits to general-purpose hetero- geneous treatment effect estimation, and we de- sign a simple and computationally efficient algo- rithm based on this reduction. Our theoretical and experimental results demonstrate that hetero- geneous treatment effect estimation in contextual bandits offers practical advantages over reward estimation, including more efficient model esti- mation and greater flexibility to model misspeci- fication.