IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/83679.html
   My bibliography  Save this paper

Cross-Validating Synthetic Controls

Author

Listed:
  • Becker, Martin
  • Klößner, Stefan
  • Pfeifer, Gregor

Abstract

While the literature on synthetic control methods mostly abstracts from out-of-sample measures, Abadie et al. (2015) have recently introduced a cross-validation approach. This technique, however, is not well-defined since it hinges on predictor weights which are not uniquely defined. We fix this issue, proposing a new, well-defined cross-validation technique, which we apply to the original Abadie et al. (2015) data. Additionally, we discuss how this new technique can be used for comparing different specifications based on out-of-sample measures, avoiding the danger of cherry-picking.

Suggested Citation

  • Becker, Martin & Klößner, Stefan & Pfeifer, Gregor, 2017. "Cross-Validating Synthetic Controls," MPRA Paper 83679, University Library of Munich, Germany.
    • Handle: RePEc:pra:mprapa:83679
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/83679/1/MPRA_paper_83679.pdf
    File Function: original version
    Download Restriction: no
    File URL: https://mpra.ub.uni-muenchen.de/83679/2/SCM_CrossValidation_2.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    2. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    3. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    4. Javier Gardeazabal & Ainhoa Vega‐Bayo, 2017. "An Empirical Comparison Between the Synthetic Control Method and HSIAO et al.'s Panel Data Approach to Program Evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(5), pages 983-1002, August.
    5. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    6. José G. Montalvo, 2011. "Voting after the Bombings: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1146-1154, November.
    7. Becker, Martin & Klößner, Stefan, 2018. "Fast and reliable computation of generalized synthetic controls," Econometrics and Statistics, Elsevier, vol. 5(C), pages 1-19.
    8. Stefan Klößner & Gregor Pfeifer, 2018. "Outside the box: using synthetic control methods as a forecasting technique," Applied Economics Letters, Taylor & Francis Journals, vol. 25(9), pages 615-618, May.
    9. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    10. Stefan Klößner & Ashok Kaul & Gregor Pfeifer & Manuel Schieler, 2018. "Comparative politics and the synthetic control method revisited: a note on Abadie et al. (2015)," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 154(1), pages 1-11, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kuosmanen, Timo & Zhou, Xun & Eskelinen, Juha & Malo, Pekka, 2021. "Design Flaw of the Synthetic Control Method," MPRA Paper 106328, University Library of Munich, Germany.
    2. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    3. Gregor Pfeifer & Fabian Wahl & Martyna Marczak, 2018. "Illuminating the World Cup effect: Night lights evidence from South Africa," Journal of Regional Science, Wiley Blackwell, vol. 58(5), pages 887-920, November.
    4. Malo, Pekka & Eskelinen, Juha & Zhou, Xun & Kuosmanen, Timo, 2020. "Computing Synthetic Controls Using Bilevel Optimization," MPRA Paper 104085, University Library of Munich, Germany.
    5. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    2. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    3. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    4. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    5. Echevarría, Cruz A. & Hasancebi, Serhat & García-Enríquez, Javier, 2022. "Economic Effects of Macao’s Integration with Mainland China: A Causal Inference Study," Journal of Economic Integration, Center for Economic Integration, Sejong University, vol. 37(2), pages 179-215.
    6. Kuosmanen, Timo & Zhou, Xun & Eskelinen, Juha & Malo, Pekka, 2021. "Design Flaw of the Synthetic Control Method," MPRA Paper 106328, University Library of Munich, Germany.
    7. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2021. "Decoupling synthetic control methods to ensure stability, accuracy and meaningfulness," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 12(4), pages 549-584, December.
    8. Essers, Dennis & Ide, Stefaan, 2019. "The IMF and precautionary lending: An empirical evaluation of the selectivity and effectiveness of the Flexible Credit Line," Journal of International Money and Finance, Elsevier, vol. 92(C), pages 25-61.
    9. Brehm, Johannes & aus dem Moore, Nils & Gruhl, Henri, 2022. "Driving Innovation? – Carbon Tax Effects in the Swedish Transport Sector," VfS Annual Conference 2022 (Basel): Big Data in Economics 264085, Verein für Socialpolitik / German Economic Association.
    10. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    11. Malo, Pekka & Eskelinen, Juha & Zhou, Xun & Kuosmanen, Timo, 2020. "Computing Synthetic Controls Using Bilevel Optimization," MPRA Paper 104085, University Library of Munich, Germany.
    12. Maximiliano Marzetti & Rok Spruk, 2023. "Long-Term Economic Effects of Populist Legal Reforms: Evidence from Argentina," Comparative Economic Studies, Palgrave Macmillan;Association for Comparative Economic Studies, vol. 65(1), pages 60-95, March.
    13. Michael Funke & Kadri Männasoo & Helery Tasane, 2023. "Regional Economic Impacts of the Øresund Cross-Border Fixed Link: Cui Bono?," CESifo Working Paper Series 10557, CESifo.
    14. Robbert Maseland & Rok Spruk, 2023. "The benefits of US statehood: an analysis of the growth effects of joining the USA," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(1), pages 49-89, January.
    15. Tomasz Serwach, 2022. "The European Union and within-country income inequalities. The case of the New Member States," Working Papers hal-03548416, HAL.
    16. Gabriel, Ricardo Duque & Pessoa, Ana Sofia, 2020. "Adopting the Euro: a synthetic control approach," MPRA Paper 99391, University Library of Munich, Germany.
    17. Roy Cerqueti & Raffaella Coppier & Alessandro Girardi & Marco Ventura, 2022. "The sooner the better: lives saved by the lockdown during the COVID-19 outbreak. The case of Italy [Using synthetic controls: Feasibility, data requirements, and methodological aspects]," The Econometrics Journal, Royal Economic Society, vol. 25(1), pages 46-70.
    18. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    19. Benjamin Krebs & Simon Luechinger, 2020. "The effect of an electricity tax on aggregate electricity consumption: evidence from Basel," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 156(1), pages 1-20, December.
    20. Pier Basaglia & Sophie Behr & Moritz A. Drupp, 2023. "De-Fueling Externalities: How Tax Salience and Fuel Substitution Mediate Climate and Health Benefits," Discussion Papers of DIW Berlin 2041, DIW Berlin, German Institute for Economic Research.

    More about this item

    Keywords

    Synthetic Control Methods; Cross-Validation; Specification Search.;
    All these keywords.

    JEL classification:

    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:83679. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.