A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study

The internal validity of a causal inference made based on an observational study is often subject to debate. The potential outcomes framework of causal inference stipulates that causal inference is essentially a missing data problem, and we follow this spirit to define the ideal sample as the combination of the observed data and the missing/counterfactual data for regression models. The robustness of a causal inference can be quantified by the probability of a robust inference for internal validity in regression, i.e., the PIVR, which is the probability of rejecting the null hypothesis again for the ideal sample provided the same null hypothesis has been already rejected for the observed sample. Drawing on the relationship between the PIVR and the mean counterfactual outcomes, we formalize a conceptual framework of quantifying the robustness of a regression-based causal inference based on a joint distribution about the mean counterfactual outcomes, holding the observed sample fixed. Interpretatively, the PIVR is the statistical power of the null hypothesis significance testing that is thought to be built on the ideal sample. We demonstrate the conceptual framework of quantifying the robustness of a regression-based causal inference with an empirical example.