IDEAS home Printed from https://ideas.repec.org/a/bpj/causin/v10y2022i1p372-414n1.html
   My bibliography  Save this article

Causal effect on a target population: A sensitivity analysis to handle missing covariates

Author

Listed:
  • Colnet Bénédicte

    (Soda Project-team, Premedical Project-team, INRIA, and Centre de Mathémathiques Appliquées, Institut Polytechnique de Paris, Palaiseau, France)

  • Josse Julie

    (Premedical Project Team, INRIA Sophia-Antipolis, Montpellier, France)

  • Varoquaux Gaël

    (Soda Project-team, INRIA Saclay, France)

  • Scornet Erwan

    (Centre de Mathémathiques Appliquées, UMR 7641, École Polytechnique, CNRS, Institut Polytechnique de Paris, Palaiseau, France)

Abstract

Randomized controlled trials (RCTs) are often considered the gold standard for estimating causal effect, but they may lack external validity when the population eligible to the RCT is substantially different from the target population. Having at hand a sample of the target population of interest allows us to generalize the causal effect. Identifying the treatment effect in the target population requires covariates to capture all treatment effect modifiers that are shifted between the two sets. Standard estimators then use either weighting (IPSW), outcome modeling (G-formula), or combine the two in doubly robust approaches (AIPSW). However, such covariates are often not available in both sets. In this article, after proving L1{L}^{1}-consistency of these three estimators, we compute the expected bias induced by a missing covariate, assuming a Gaussian distribution, a continuous outcome, and a semi-parametric model. Under this setting, we perform a sensitivity analysis for each missing covariate pattern and compute the sign of the expected bias. We also show that there is no gain in linearly imputing a partially unobserved covariate. Finally, we study the substitution of a missing covariate by a proxy. We illustrate all these results on simulations, as well as semi-synthetic benchmarks using data from the Tennessee student/teacher achievement ratio (STAR), and a real-world example from critical care medicine.

Suggested Citation

  • Colnet Bénédicte & Josse Julie & Varoquaux Gaël & Scornet Erwan, 2022. "Causal effect on a target population: A sensitivity analysis to handle missing covariates," Journal of Causal Inference, De Gruyter, vol. 10(1), pages 372-414, January.
  • Handle: RePEc:bpj:causin:v:10:y:2022:i:1:p:372-414:n:1
    DOI: 10.1515/jci-2021-0059
    as

    Download full text from publisher

    File URL: https://doi.org/10.1515/jci-2021-0059
    Download Restriction: no

    File URL: https://libkey.io/10.1515/jci-2021-0059?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    2. Andrea Ichino & Fabrizia Mealli & Tommaso Nannicini, 2008. "From temporary help jobs to permanent employment: what can we learn from matching estimators and their sensitivity?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 23(3), pages 305-327.
    3. Guildo W. Imbens, 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation," American Economic Review, American Economic Association, vol. 93(2), pages 126-132, May.
    4. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    5. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    6. Carlos Cinelli & Chad Hazlett, 2020. "Making sense of sensitivity: extending omitted variable bias," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(1), pages 39-67, February.
    7. Issa J. Dahabreh & Sarah E. Robertson & Eric J. Tchetgen & Elizabeth A. Stuart & Miguel A. Hernán, 2019. "Generalizing causal inferences from individuals in randomized trials to all trial‐eligible individuals," Biometrics, The International Biometric Society, vol. 75(2), pages 685-694, June.
    8. Xinkun Nie & Guido Imbens & Stefan Wager, 2021. "Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations," Papers 2112.04723, arXiv.org.
    9. Xiaohong Chen & Han Hong & Elie Tamer, 2005. "Measurement Error Models with Auxiliary Data," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(2), pages 343-366.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Paul Diegert & Matthew A. Masten & Alexandre Poirier, 2022. "Assessing Omitted Variable Bias when the Controls are Endogenous," Papers 2206.02303, arXiv.org, revised Jul 2023.
    2. Matthew A. Masten & Alexandre Poirier & Linqi Zhang, 2024. "Assessing Sensitivity to Unconfoundedness: Estimation and Inference," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(1), pages 1-13, January.
    3. Hao, Shiming, 2021. "True structure change, spurious treatment effect? A novel approach to disentangle treatment effects from structure changes," MPRA Paper 108679, University Library of Munich, Germany.
    4. Victor Chernozhukov & Carlos Cinelli & Whitney Newey & Amit Sharma & Vasilis Syrgkanis, 2021. "Long Story Short: Omitted Variable Bias in Causal Machine Learning," Papers 2112.13398, arXiv.org, revised Nov 2023.
    5. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    6. repec:zbw:rwirep:0426 is not listed on IDEAS
    7. Oliveira, Rodrigo & Moura, Klebson & Viana, Jorge & Tigre, Robson & Sampaio, Breno, 2015. "Commute duration and health: Empirical evidence from Brazil," Transportation Research Part A: Policy and Practice, Elsevier, vol. 80(C), pages 62-75.
    8. Hendrik Schmitz & Matthias Westphal, 2013. "Short- and Medium-term Effects of Informal Care Provision on Health," Ruhr Economic Papers 0426, Rheinisch-Westfälisches Institut für Wirtschaftsforschung, Ruhr-Universität Bochum, Universität Dortmund, Universität Duisburg-Essen.
    9. Matthew Masten & Alexandre Poirier, 2016. "Partial independence in nonseparable models," CeMMAP working papers CWP26/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    10. Bo Zhang & Dylan S. Small, 2020. "A calibrated sensitivity analysis for matched observational studies with application to the effect of second‐hand smoke exposure on blood lead levels in children," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1285-1305, November.
    11. Bo Zhang & Eric J. Tchetgen Tchetgen, 2022. "A semi‐parametric approach to model‐based sensitivity analysis in observational studies," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(S2), pages 668-691, December.
    12. Naoki Egami & Erin Hartman, 2021. "Covariate selection for generalizing experimental results: Application to a large‐scale development program in Uganda," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(4), pages 1524-1548, October.
    13. Paolo Naticchioni & Silvia Loriga, 2011. "Short and Long Term Evaluations of Public Employment Services in Italy," Applied Economics Quarterly (formerly: Konjunkturpolitik), Duncker & Humblot, Berlin, vol. 57(3), pages 201-229.
    14. Marie Bjørneby & Annette Alstadsæter & Kjetil Telle, 2018. "Collusive tax evasion by employers and employees. Evidence from a randomized fi eld experiment in Norway," Discussion Papers 891, Statistics Norway, Research Department.
    15. Pedro H. C. Sant'Anna & Xiaojun Song & Qi Xu, 2022. "Covariate distribution balance via propensity scores," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(6), pages 1093-1120, September.
    16. Dennis Shen & Peng Ding & Jasjeet Sekhon & Bin Yu, 2022. "Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data," Papers 2207.14481, arXiv.org, revised Oct 2022.
    17. Art B. Owen & Hal Varian, 2018. "Optimizing the tie-breaker regression discontinuity design," Papers 1808.07563, arXiv.org, revised Jul 2020.
    18. Cristian Mardones & Pablo Herreros, 2023. "Ex post evaluation of voluntary environmental policies on the energy intensity in Chilean firms," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 25(9), pages 9111-9136, September.
    19. Sloczynski, Tymon, 2020. "Interpreting OLS Estimands When Treatment Effects Are Heterogeneous: Smaller Groups Get Larger Weights," IZA Discussion Papers 13283, Institute of Labor Economics (IZA).
    20. Tommaso Nannicini, 2007. "Simulation-based sensitivity analysis for matching estimators," Stata Journal, StataCorp LP, vol. 7(3), pages 334-350, September.
    21. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:causin:v:10:y:2022:i:1:p:372-414:n:1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.