IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/78079.html
   My bibliography  Save this paper

Placebo Tests for Synthetic Controls

Author

Listed:
  • Ferman, Bruno
  • Pinto, Cristine

Abstract

The synthetic control (SC) method has been recently proposed as an alternative to estimate treatment effects in comparative case studies. An important feature of the SC method is the inferential procedures based on placebo studies, suggested in Abadie et al. (2010). In this paper, we evaluate the statistical properties of these inferential techniques. We first show that the graphical analysis with placebos can be misleading, as placebo runs with lower expected squared prediction errors would still be considered in the analysis. Then we show that a test based on the the post/pre-intervention mean squared prediction error, as suggested in Abadie et al. (2010), ameliorates this problem. However, we show that such test can still have some size distortions, even if we consider a case in which the test statistic has the same marginal distribution for all placebo runs. Finally, we show that the fact that the SC weights are estimated can lead to important additional size distortions.

Suggested Citation

  • Ferman, Bruno & Pinto, Cristine, 2017. "Placebo Tests for Synthetic Controls," MPRA Paper 78079, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:78079
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/78079/1/MPRA_paper_78079.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bruno Ferman & Cristine Pinto, 2019. "Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity," The Review of Economics and Statistics, MIT Press, vol. 101(3), pages 452-467, July.
    2. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    3. repec:fgv:eesptd:411 is not listed on IDEAS
    4. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    5. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    6. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    7. Carlos Viana de Carvalho & Ricardo Masini & Marcelo Cunha Medeiros, 2016. "The perils of Counterfactual Analysis with Integrated Processes," Textos para discussão 654, Department of Economics PUC-Rio (Brazil).
    8. Ferman, Bruno & Pinto, Cristine Campos de Xavier, 2016. "Revisiting the synthetic control estimator," Textos para discussão 421, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    9. David Powell, 2016. "Synthetic Control Estimation Beyond Case Studies Does the Minimum Wage Reduce Employment?," Working Papers 1142, RAND Corporation.
    10. Jinyong Hahn & Ruoyao Shi, 2017. "Synthetic Control and Inference," Econometrics, MDPI, Open Access Journal, vol. 5(4), pages 1-12, November.
    11. David Powell, 2016. "Synthetic Control Estimation Beyond Case Studies Does the Minimum Wage Reduce Employment?," Working Papers WR-1142, RAND Corporation.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Felipe Filgueiras, Elias Cavalcante-Filho, Rodrigo de Losso, José Roberto Savoia, 2019. "Law Change in a Regulated Sector Impacts Other Regulated Sectors: Evidence from Brazil," Working Papers, Department of Economics 2019_27, University of São Paulo (FEA-USP).
    2. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    3. Bluszcz, Julia & Valente, Marica, 2020. "The Economic Costs of Hybrid Wars: The Case of Ukraine," EconStor Open Access Articles, ZBW - Leibniz Information Centre for Economics, pages 1-25.
    4. Benjamin Krebs & Simon Luechinger, 2020. "The effect of an electricity tax on aggregate electricity consumption: evidence from Basel," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 156(1), pages 1-20, December.
    5. Ferman, Bruno, 2017. "Matching Estimators with Few Treated and Many Control Observations," MPRA Paper 78940, University Library of Munich, Germany.
    6. Andre Gbato & Falapalaki Lemou & Jean-François Brun, 2021. "Effectiveness of SARA reform in sub-Saharan Africa [Efficacité de la réforme des SARA en Afrique subsaharienne]," Working Papers hal-03119001, HAL.
    7. Mellace, Giovanni & Pasquini, Alessandra, 2019. "Identify More, Observe Less: Mediation Analysis: Mediation Analysis Synthetic Control," Discussion Papers of Business and Economics 12/2019, University of Southern Denmark, Department of Business and Economics.
    8. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    9. Lea Bottmer & Guido Imbens & Jann Spiess & Merrill Warnick, 2021. "A Design-Based Perspective on Synthetic Control Methods," Papers 2101.09398, arXiv.org.
    10. Julia Bluszcz & Marica Valente, 2019. "The War in Europe: Economic Costs of the Ukrainian Conflict," Discussion Papers of DIW Berlin 1804, DIW Berlin, German Institute for Economic Research.
    11. Jianfei Cao & Connor Dowd, 2019. "Estimation and Inference for Synthetic Control Methods with Spillover Effects," Papers 1902.07343, arXiv.org, revised Nov 2019.
    12. Nadler, Carl & Allegretto, Sylvia & Godoey, Anna & Reich, Michael, 2019. "Are Local Minimum Wages Too High? Working Paper #102-19," Institute for Research on Labor and Employment, Working Paper Series qt7xt8716f, Institute of Industrial Relations, UC Berkeley.
    13. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2019. "Inference on average treatment effects in aggregate panel data settings," CeMMAP working papers CWP32/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    14. Peri, Giovanni & Rury, Derek & Wiltshire, Justin C., 2020. "The Economic Impact of Migrants from Hurricane Maria," IZA Discussion Papers 13049, Institute of Labor Economics (IZA).
    15. Victor Chernozhukov & Kaspar Wüthrich & Yu Zhu, 2017. "An exact and robust conformal inference method for counterfactual and synthetic controls," CeMMAP working papers CWP62/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    16. Victor Chernozhukov & Kaspar Wuthrich & Yinchu Zhu, 2018. "A $t$-test for synthetic controls," Papers 1812.10820, arXiv.org, revised Apr 2021.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    2. Carvalho, Carlos & Masini, Ricardo & Medeiros, Marcelo C., 2018. "ArCo: An artificial counterfactual approach for high-dimensional panel time-series data," Journal of Econometrics, Elsevier, vol. 207(2), pages 352-380.
    3. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    4. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    5. Ferman, Bruno, 2017. "Matching Estimators with Few Treated and Many Control Observations," MPRA Paper 78940, University Library of Munich, Germany.
    6. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    7. Muhammad Jehangir Amjad & Devavrat Shah & Dennis Shen, 2017. "Robust Synthetic Control," Papers 1711.06940, arXiv.org.
    8. Jason Poulos & Shuxi Zeng, 2017. "RNN-based counterfactual prediction, with an application to homestead policy and public schooling," Papers 1712.03553, arXiv.org, revised Sep 2020.
    9. Yi‐Ting Chen, 2020. "A distributional synthetic control method for policy evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 505-525, August.
    10. Kuosmanen, Timo & Zhou, Xun & Eskelinen, Juha & Malo, Pekka, 2021. "Design Flaw of the Synthetic Control Method," MPRA Paper 106328, University Library of Munich, Germany.
    11. Gabriel, Ricardo Duque & Pessoa, Ana Sofia, 2020. "Adopting the Euro: a synthetic control approach," MPRA Paper 99391, University Library of Munich, Germany.
    12. Giovanni Peri & Derek Rury & Justin C. Wiltshire, 2020. "The Economic Impact of Migrants from Hurricane Maria," NBER Working Papers 27718, National Bureau of Economic Research, Inc.
    13. Giulio Grossi & Patrizia Lattarulo & Marco Mariani & Alessandra Mattei & Ozge Oner, 2020. "Synthetic Control Group Methods in the Presence of Interference: The Direct and Spillover Effects of Light Rail on Neighborhood Retail Activity," Papers 2004.05027, arXiv.org, revised Feb 2021.
    14. Roy Cerqueti & Raffaella Coppier & Alessandro Girardi & Marco Ventura, 2021. "The sooner the better: lives saved by the lockdown during the COVID-19 outbreak. The case of Italy," Papers 2101.11901, arXiv.org.
    15. Bruno Ferman, 2019. "On the Properties of the Synthetic Control Estimator with Many Periods and Many Controls," Papers 1906.06665, arXiv.org, revised May 2020.
    16. Felipe Filgueiras, Elias Cavalcante-Filho, Rodrigo de Losso, José Roberto Savoia, 2019. "Law Change in a Regulated Sector Impacts Other Regulated Sectors: Evidence from Brazil," Working Papers, Department of Economics 2019_27, University of São Paulo (FEA-USP).
    17. Martin Becker & Stefan Klößner & Gregor Pfeifer, 2018. "Cross-Validating Synthetic Controls," Economics Bulletin, AccessEcon, vol. 38(1), pages 603-609.
    18. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2017. "Matrix Completion Methods for Causal Panel Data Models," Papers 1710.10251, arXiv.org, revised Feb 2021.
    19. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    20. Victor Chernozhukov & Kaspar Wuthrich & Yinchu Zhu, 2017. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Papers 1712.09089, arXiv.org, revised Sep 2020.

    More about this item

    Keywords

    synthetic control; difference-in-differences; linear factor model; inference; permutation test;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:78079. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Joachim Winter). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.