IDEAS home Printed from https://ideas.repec.org/p/fgv/eesptd/420.html
   My bibliography  Save this paper

Cherry picking with synthetic controls

Author

Listed:
  • Ferman, Bruno
  • Pinto, Cristine Campos de Xavier
  • Possebom, Vítor Augusto

Abstract

The synthetic control (SC) method has been recently proposed as an alternative method to estimate treatment e ects in comparative case studies. Abadie et al. [2010] and Abadie et al. [2015] argue that one of the advantages of the SC method is that it imposes a data-driven process to select the comparison units, providing more transparency and less discretionary power to the researcher. However, an important limitation of the SC method is that it does not provide clear guidance on the choice of predictor variables used to estimate the SC weights. We show that such lack of speci c guidances provides signi cant opportunities for the researcher to search for speci cations with statistically signi cant results, undermining one of the main advantages of the method. Considering six alternative speci cations commonly used in SC applications, we calculate in Monte Carlo simulations the probability of nding a statistically signi cant result at 5% in at least one speci cation. We nd that this probability can be as high as 13% (23% for a 10% signi cance test) when there are 12 pre-intervention periods and decay slowly with the number of pre-intervention periods. With 230 pre-intervention periods, this probability is still around 10% (18% for a 10% signi cance test). We show that the speci cation that uses the average pre-treatment outcome values to estimate the weights performed particularly bad in our simulations. However, the speci cation-searching problem remains relevant even when we do not consider this speci cation. We also show that this speci cation-searching problem is relevant in simulations with real datasets looking at placebo interventions in the Current Population Survey (CPS). In order to mitigate this problem, we propose a criterion to select among SC di erent speci cations based on the prediction error of each speci cations in placebo estimations

Suggested Citation

  • Ferman, Bruno & Pinto, Cristine Campos de Xavier & Possebom, Vítor Augusto, 2016. "Cherry picking with synthetic controls," Textos para discussão 420, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
  • Handle: RePEc:fgv:eesptd:420
    as

    Download full text from publisher

    File URL: http://bibliotecadigital.fgv.br/dspace/bitstream/10438/16583/1/TD%20420%20-%20BrunoFerman%20e%20CristinePinto%20e%20Vitor.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Isabela Ferreira Duarte & João Manoel Pinho de Mello & Vinicius Nascimento Carrasco, 2014. "A Década Perdida: 2003 – 2012," Textos para discussão 626, Department of Economics PUC-Rio (Brazil).
    2. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    3. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    4. Gabriela Calderón, 2014. "The Effects of Child Care Provision in Mexico," Working Papers 2014-07, Banco de México.
    5. José G. Montalvo, 2011. "Voting after the Bombings: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1146-1154, November.
    6. Vincenzo Bove & Leandro Eliay & Ron P Smith, 2014. "The relationship between panel and synthetic control estimators of the effect of civil war," BCAM Working Papers 1406, Birkbeck Centre for Applied Macroeconomics.
    7. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    8. Andreas Billmeier & Tommaso Nannicini, 2009. "Trade Openness and Growth: Pursuing Empirical Glasnost," IMF Staff Papers, Palgrave Macmillan, vol. 56(3), pages 447-475, August.
    9. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    10. Ando, Michihito, 2015. "Dreams of urbanization: Quantitative case studies on the local impacts of nuclear power facilities using the synthetic control method," Journal of Urban Economics, Elsevier, vol. 85(C), pages 68-85.
    11. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    12. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    13. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, Southern Economic Association, vol. 83(2), pages 437-463, October.
    14. Eduardo Cavallo & Sebastian Galiani & Ilan Noy & Juan Pantano, 2013. "Catastrophic Natural Disasters and Economic Growth," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1549-1561, December.
    15. repec:wly:hlthec:v:25:y:2016:i:12:p:1514-1528 is not listed on IDEAS
    16. Marcos Sanso‐Navarro, 2011. "The Effects on American Foreign Direct Investment in the United Kingdom from Not Adopting the Euro," Journal of Common Market Studies, Wiley Blackwell, vol. 49(2), pages 463-483, March.
    17. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    18. repec:spr:sjecst:v:154:y:2018:i:1:d:10.1186_s41937-017-0004-9 is not listed on IDEAS
    19. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    20. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    21. Jinjarak, Yothin & Noy, Ilan & Zheng, Huanhuan, 2013. "Capital controls in Brazil – Stemming a tide with a signal?," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2938-2952.
    22. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    23. Henrik Jacobsen Kleven & Camille Landais & Emmanuel Saez, 2013. "Taxation and International Migration of Superstars: Evidence from the European Football Market," American Economic Review, American Economic Association, vol. 103(5), pages 1892-1924, August.
    24. Barone, Guglielmo & Mocetti, Sauro, 2014. "Natural disasters, growth and institutions: A tale of two earthquakes," Journal of Urban Economics, Elsevier, vol. 84(C), pages 52-66.
    25. Amr Hosny, 2012. "Algeria’s Trade with GAFTA Countries: A Synthetic Control Approach," Transition Studies Review, Springer;Central Eastern European University Network (CEEUN), vol. 19(1), pages 35-42, September.
    26. Smith, Brock, 2015. "The resource curse exorcised: Evidence from a panel of countries," Journal of Development Economics, Elsevier, vol. 116(C), pages 57-73.
    27. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2011. "Synth: An R Package for Synthetic Control Methods in Comparative Case Studies," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i13).
    28. Shimeng Liu, 2015. "Spillovers from Universities: Evidence from the Land-Grant Program," Working Paper 9410, USC Lusk Center for Real Estate.
    29. William duPont IV & Ilan Noy, 2015. "What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan," Economic Development and Cultural Change, University of Chicago Press, vol. 63(4), pages 777-812.
    30. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    31. repec:aea:jeclit:v:56:y:2018:i:3:p:920-80 is not listed on IDEAS
    32. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    33. Bauhoff, Sebastian, 2014. "The effect of school district nutrition policies on dietary intake and overweight: A synthetic control approach," Economics & Human Biology, Elsevier, vol. 12(C), pages 45-55.
    34. Liu, Shimeng, 2015. "Spillovers from universities: Evidence from the land-grant program," Journal of Urban Economics, Elsevier, vol. 87(C), pages 25-41.
    35. Ferman, Bruno & Pinto, Cristine Campos de Xavier, 2016. "Revisiting the synthetic control estimator," Textos para discussão 421, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    36. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    37. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    38. Mideksa, Torben K., 2013. "The economic impact of natural resources," Journal of Environmental Economics and Management, Elsevier, vol. 65(2), pages 277-289.
    39. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    40. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    41. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    42. Severnini, Edson R., 2014. "The Power of Hydroelectric Dams: Agglomeration Spillovers," IZA Discussion Papers 8082, Institute of Labor Economics (IZA).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Acel Jardón & Onno Kuik & Richard S.J. Tol, 2018. "Causal effects of PetroCaribe on sustainable development: a synthetic control analysis," Working Paper Series 0918, Department of Economics, University of Sussex Business School.
    2. repec:eee:econom:v:207:y:2018:i:2:p:352-380 is not listed on IDEAS
    3. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    4. Botosaru, Irene & Ferman, Bruno, 2017. "On the Role of Covariates in the Synthetic Control Method," MPRA Paper 80796, University Library of Munich, Germany.
    5. repec:ebl:ecbull:eb-17-00821 is not listed on IDEAS
    6. Martin Becker & Stefan Klößner & Gregor Pfeifer, 2018. "Cross-Validating Synthetic Controls," Economics Bulletin, AccessEcon, vol. 38(1), pages 603-609.
    7. Muhammad Jehangir Amjad & Devavrat Shah & Dennis Shen, 2017. "Robust Synthetic Control," Papers 1711.06940, arXiv.org.
    8. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    9. Carvalho, Carlos & Masini, Ricardo & Medeiros, Marcelo C., 2018. "ArCo: An artificial counterfactual approach for high-dimensional panel time-series data," Journal of Econometrics, Elsevier, vol. 207(2), pages 352-380.
    10. Roel Dom, 2017. "Semi-Autonomous Revenue Authorities in Sub-Saharan Africa: Silver Bullet or White Elephant," Discussion Papers 2017-01, University of Nottingham, CREDIT.
    11. Jason Poulos, 2017. "RNN-based counterfactual prediction," Papers 1712.03553, arXiv.org, revised Apr 2019.
    12. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    13. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fgv:eesptd:420. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Núcleo de Computação da FGV EPGE). General contact details of provider: http://edirc.repec.org/data/eegvfbr.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.