IDEAS home Printed from
   My bibliography  Save this paper

An exact and robust conformal inference method for counterfactual and synthetic controls


  • Victor Chernozhukov

    () (Institute for Fiscal Studies and MIT)

  • Kaspar Wüthrich

    (Institute for Fiscal Studies and UCSD)

  • Yu Zhu

    (Institute for Fiscal Studies)


This paper introduces new inference methods for counterfactual and synthetic control methods for evaluating policy effects. Our inference methods work in conjunction with many modern and classical methods for estimating the counterfactual mean outcome in the absence of a policy intervention. Specifically, our methods work together with the difference-in-difference, canonical synthetic control, constrained and penalized regression methods for synthetic control, factor/matrix completion models for panel data, interactive fixed effects panel models, time series models, as well as fused time series panel data models. The proposed method has a double justification. (i) If the residuals from estimating the counterfactuals are exchangeable as implied, for example, by i.i.d. data, our procedure achieves exact finite sample size control without any assumption on the specific approach used to estimate the counterfactuals. (ii) If the data exhibit dynamics and serial dependence, our inference procedure achieves approximate uniform size control under weak and easy-to-verify conditions on the method used to estimate the counterfactual. We verify these condition for representative methods from each group listed above. Simulation experiments demonstrate the usefulness of our approach in finite samples. We apply our method to re-evaluate the causal effect of election day registration (EDR) laws on voter turnout in the United States.

Suggested Citation

  • Victor Chernozhukov & Kaspar Wüthrich & Yu Zhu, 2017. "An exact and robust conformal inference method for counterfactual and synthetic controls," CeMMAP working papers CWP62/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  • Handle: RePEc:ifs:cemmap:62/17

    Download full text from publisher

    File URL:
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. repec:tpr:restat:v:101:y:2019:i:3:p:452-467 is not listed on IDEAS
    2. Hansen, Christian & Liao, Yuan, 2019. "The Factor-Lasso And K-Step Bootstrap Approach For Inference In High-Dimensional Economic Applications," Econometric Theory, Cambridge University Press, vol. 35(03), pages 465-509, June.
    3. M. Hashem Pesaran, 2006. "Estimation and Inference in Large Heterogeneous Panels with a Multifactor Error Structure," Econometrica, Econometric Society, vol. 74(4), pages 967-1012, July.
    4. D. W. K. Andrews, 2003. "End-of-Sample Instability Tests," Econometrica, Econometric Society, vol. 71(6), pages 1661-1694, November.
    5. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    6. Bruno Ferman & Cristine Pinto, 2019. "Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 452-467, July.
    7. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    8. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 605-654.
    9. Timothy G. Conley & Christopher R. Taber, 2011. "Inference with "Difference in Differences" with a Small Number of Policy Changes," The Review of Economics and Statistics, MIT Press, vol. 93(1), pages 113-125, February.
    10. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2017. "Matrix Completion Methods for Causal Panel Data Models," Papers 1710.10251,, revised Sep 2018.
    11. Christian Hansen & Yuan Liao, 2016. "The Factor-Lasso and K-Step Bootstrap Approach for Inference in High-Dimensional Economic Applications," Departmental Working Papers 201610, Rutgers University, Department of Economics.
    12. repec:bpj:causin:v:6:y:2018:i:2:p:26:n:1 is not listed on IDEAS
    13. Christian Hansen & Yuan Liao, 2016. "The Factor-Lasso and K-Step Bootstrap Approach for Inference in High-Dimensional Economic Applications," Papers 1611.09420,, revised Dec 2016.
    14. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    15. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    16. Muhammad Jehangir Amjad & Devavrat Shah & Dennis Shen, 2017. "Robust Synthetic Control," Papers 1711.06940,
    17. Ashenfelter, Orley & Card, David, 1985. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs," The Review of Economics and Statistics, MIT Press, vol. 67(4), pages 648-660, November.
    18. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    19. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    20. Giovanni Peri & Vasil Yasenov, 2015. "The Labor Market Effects of a Refugee Wave: Applying the Synthetic Control Method to the Mariel Boatlift," NBER Working Papers 21801, National Bureau of Economic Research, Inc.
    21. Carvalho, Carlos & Masini, Ricardo & Medeiros, Marcelo C., 2018. "ArCo: An artificial counterfactual approach for high-dimensional panel time-series data," Journal of Econometrics, Elsevier, vol. 207(2), pages 352-380.
    22. Ferman, Bruno & Pinto, Cristine, 2017. "Placebo Tests for Synthetic Controls," MPRA Paper 78079, University Library of Munich, Germany.
    23. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    24. Jing Lei & James Robins & Larry Wasserman, 2013. "Distribution-Free Prediction Sets," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(501), pages 278-287, March.
    25. Xu, Yiqing, 2017. "Generalized Synthetic Control Method: Causal Inference with Interactive Fixed Effects Models," Political Analysis, Cambridge University Press, vol. 25(01), pages 57-76, January.
    26. Jushan Bai, 2009. "Panel Data Models With Interactive Fixed Effects," Econometrica, Econometric Society, vol. 77(4), pages 1229-1279, July.
    27. Dukpa Kim & Tatsushi Oka, 2014. "Divorce Law Reforms And Divorce Rates In The Usa: An Interactive Fixed‐Effects Approach," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(2), pages 231-245, March.
    28. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    29. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    30. Jushan Bai, 2003. "Inferential Theory for Factor Models of Large Dimensions," Econometrica, Econometric Society, vol. 71(1), pages 135-171, January.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Jianfei Cao & Connor Dowd, 2019. "Estimation and Inference for Synthetic Control Methods with Spillover Effects," Papers 1902.07343,
    2. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490,
    3. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2018. "Exact and robust conformal inference methods for predictive machine learning with dependent data," CeMMAP working papers CWP16/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    4. Ferman, Bruno, 2017. "Matching Estimators with Few Treated and Many Control Observations," MPRA Paper 78940, University Library of Munich, Germany.
    5. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    6. Victor Chernozhukov & Kaspar Wuthrich & Yinchu Zhu, 2018. "Practical and robust $t$-test based inference for synthetic control and related methods," Papers 1812.10820,, revised Jun 2019.
    7. Bruno Ferman, 2019. "On the Properties of the Synthetic Control Estimator with Many Periods and Many Controls," Papers 1906.06665,, revised Oct 2019.
    8. Jason Poulos, 2019. "State-Building through Public Land Disposal? An Application of Matrix Completion for Counterfactual Prediction," Papers 1903.08028,
    9. Eli Ben-Michael & Avi Feller & Jesse Rothstein, 2018. "The Augmented Synthetic Control Method," Papers 1811.04170,

    More about this item


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ifs:cemmap:62/17. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Emma Hyman). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.