IDEAS home Printed from https://ideas.repec.org/p/fgv/eesptd/420.html
   My bibliography  Save this paper

Cherry picking with synthetic controls

Author

Listed:
  • Ferman, Bruno
  • Pinto, Cristine Campos de Xavier
  • Possebom, Vítor Augusto

Abstract

The synthetic control (SC) method has been recently proposed as an alternative method to estimate treatment e ects in comparative case studies. Abadie et al. [2010] and Abadie et al. [2015] argue that one of the advantages of the SC method is that it imposes a data-driven process to select the comparison units, providing more transparency and less discretionary power to the researcher. However, an important limitation of the SC method is that it does not provide clear guidance on the choice of predictor variables used to estimate the SC weights. We show that such lack of speci c guidances provides signi cant opportunities for the researcher to search for speci cations with statistically signi cant results, undermining one of the main advantages of the method. Considering six alternative speci cations commonly used in SC applications, we calculate in Monte Carlo simulations the probability of nding a statistically signi cant result at 5% in at least one speci cation. We nd that this probability can be as high as 13% (23% for a 10% signi cance test) when there are 12 pre-intervention periods and decay slowly with the number of pre-intervention periods. With 230 pre-intervention periods, this probability is still around 10% (18% for a 10% signi cance test). We show that the speci cation that uses the average pre-treatment outcome values to estimate the weights performed particularly bad in our simulations. However, the speci cation-searching problem remains relevant even when we do not consider this speci cation. We also show that this speci cation-searching problem is relevant in simulations with real datasets looking at placebo interventions in the Current Population Survey (CPS). In order to mitigate this problem, we propose a criterion to select among SC di erent speci cations based on the prediction error of each speci cations in placebo estimations

Suggested Citation

  • Ferman, Bruno & Pinto, Cristine Campos de Xavier & Possebom, Vítor Augusto, 2016. "Cherry picking with synthetic controls," Textos para discussão 420, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
  • Handle: RePEc:fgv:eesptd:420
    as

    Download full text from publisher

    File URL: http://bibliotecadigital.fgv.br/dspace/bitstream/10438/16583/1/TD%20420%20-%20BrunoFerman%20e%20CristinePinto%20e%20Vitor.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Kirkpatrick, A. Justin & Bennear, Lori S., 2014. "Promoting clean energy investment: An empirical analysis of property assessed clean energy," Journal of Environmental Economics and Management, Elsevier, vol. 68(2), pages 357-375.
    2. Isabela Ferreira Duarte & João Manoel Pinho de Mello & Vinicius Nascimento Carrasco, 2014. "A Década Perdida: 2003 – 2012," Textos para discussão 626, Department of Economics PUC-Rio (Brazil).
    3. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    4. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    5. Gabriela Calderón, 2014. "The Effects of Child Care Provision in Mexico," Working Papers 2014-07, Banco de México.
    6. José G. Montalvo, 2011. "Voting after the Bombings: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1146-1154, November.
    7. Christian Dustmann & Uta Schönberg & Jan Stuhler, 2017. "Labor Supply Shocks, Native Wages, and the Adjustment of Local Employment," The Quarterly Journal of Economics, Oxford University Press, vol. 132(1), pages 435-483.
    8. Vincenzo Bove & Leandro Eliay & Ron P Smith, 2014. "The relationship between panel and synthetic control estimators of the effect of civil war," BCAM Working Papers 1406, Birkbeck Centre for Applied Macroeconomics.
    9. Ozkan Eren & Serkan Ozbeklik, 2016. "What Do Right‐to‐Work Laws Do? Evidence from a Synthetic Control Method Analysis," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 35(1), pages 173-194, January.
    10. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    11. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    12. Andreas Billmeier & Tommaso Nannicini, 2009. "Trade Openness and Growth: Pursuing Empirical Glasnost," IMF Staff Papers, Palgrave Macmillan, vol. 56(3), pages 447-475, August.
    13. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    14. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    15. Scott Cunningham & Manisha Shah, 2018. "Decriminalizing Indoor Prostitution: Implications for Sexual Violence and Public Health," Review of Economic Studies, Oxford University Press, vol. 85(3), pages 1683-1715.
    16. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, Southern Economic Association, vol. 83(2), pages 437-463, October.
    17. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    18. Ando, Michihito, 2015. "Dreams of urbanization: Quantitative case studies on the local impacts of nuclear power facilities using the synthetic control method," Journal of Urban Economics, Elsevier, vol. 85(C), pages 68-85.
    19. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    20. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    21. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    22. Eduardo Cavallo & Sebastian Galiani & Ilan Noy & Juan Pantano, 2013. "Catastrophic Natural Disasters and Economic Growth," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1549-1561, December.
    23. Dube, Arindrajit & Zipperer, Ben, 2015. "Pooling Multiple Case Studies Using Synthetic Controls: An Application to Minimum Wage Policies," IZA Discussion Papers 8944, Institute of Labor Economics (IZA).
    24. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    25. Ferman, Bruno & Pinto, Cristine, 2017. "Placebo Tests for Synthetic Controls," MPRA Paper 78079, University Library of Munich, Germany.
    26. Ben Zou, 2018. "The Local Economic Impacts of Military Personnel," Journal of Labor Economics, University of Chicago Press, vol. 36(3), pages 589-621.
    27. Marcos Sanso‐Navarro, 2011. "The Effects on American Foreign Direct Investment in the United Kingdom from Not Adopting the Euro," Journal of Common Market Studies, Wiley Blackwell, vol. 49(2), pages 463-483, March.
    28. Stefan Klößner & Ashok Kaul & Gregor Pfeifer & Manuel Schieler, 2018. "Comparative politics and the synthetic control method revisited: a note on Abadie et al. (2015)," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 154(1), pages 1-11, December.
    29. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    30. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    31. Jinjarak, Yothin & Noy, Ilan & Zheng, Huanhuan, 2013. "Capital controls in Brazil – Stemming a tide with a signal?," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2938-2952.
    32. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    33. Henrik Jacobsen Kleven & Camille Landais & Emmanuel Saez, 2013. "Taxation and International Migration of Superstars: Evidence from the European Football Market," American Economic Review, American Economic Association, vol. 103(5), pages 1892-1924, August.
    34. Gregory DeAngelo & Benjamin Hansen, 2014. "Life and Death in the Fast Lane: Police Enforcement and Traffic Fatalities," American Economic Journal: Economic Policy, American Economic Association, vol. 6(2), pages 231-257, May.
    35. Barone, Guglielmo & Mocetti, Sauro, 2014. "Natural disasters, growth and institutions: A tale of two earthquakes," Journal of Urban Economics, Elsevier, vol. 84(C), pages 52-66.
    36. Amr Hosny, 2012. "Algeria’s Trade with GAFTA Countries: A Synthetic Control Approach," Transition Studies Review, Springer;Central Eastern European University Network (CEEUN), vol. 19(1), pages 35-42, September.
    37. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    38. Smith, Brock, 2015. "The resource curse exorcised: Evidence from a panel of countries," Journal of Development Economics, Elsevier, vol. 116(C), pages 57-73.
    39. Leonardo Baccini & Quan Li & Irina Mirkina, 2014. "Corporate Tax Cuts And Foreign Direct Investment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(4), pages 977-1006, September.
    40. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    41. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2011. "Synth: An R Package for Synthetic Control Methods in Comparative Case Studies," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i13).
    42. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    43. Shimeng Liu, 2015. "Spillovers from Universities: Evidence from the Land-Grant Program," Working Paper 9410, USC Lusk Center for Real Estate.
    44. William duPont IV & Ilan Noy, 2015. "What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan," Economic Development and Cultural Change, University of Chicago Press, vol. 63(4), pages 777-812.
    45. Firpo Sergio & Possebom Vitor, 2018. "Synthetic Control Method: Inference, Sensitivity Analysis and Confidence Sets," Journal of Causal Inference, De Gruyter, vol. 6(2), pages 1-26, September.
    46. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    47. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    48. Ann P. Bartel & Maya Rossin†Slater & Christopher J. Ruhm & Jenna Stearns & Jane Waldfogel, 2018. "Paid Family Leave, Fathers’ Leave†Taking, and Leave†Sharing in Dual†Earner Households," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 37(1), pages 10-37, January.
    49. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    50. Bauhoff, Sebastian, 2014. "The effect of school district nutrition policies on dietary intake and overweight: A synthetic control approach," Economics & Human Biology, Elsevier, vol. 12(C), pages 45-55.
    51. Liu, Shimeng, 2015. "Spillovers from universities: Evidence from the land-grant program," Journal of Urban Economics, Elsevier, vol. 87(C), pages 25-41.
    52. Ferman, Bruno & Pinto, Cristine Campos de Xavier, 2016. "Revisiting the synthetic control estimator," Textos para discussão 421, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    53. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    54. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    55. Mideksa, Torben K., 2013. "The economic impact of natural resources," Journal of Environmental Economics and Management, Elsevier, vol. 65(2), pages 277-289.
    56. Jinyong Hahn & Ruoyao Shi, 2017. "Synthetic Control and Inference," Econometrics, MDPI, Open Access Journal, vol. 5(4), pages 1-12, November.
    57. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    58. Fafchamps, Marcel & Labonne, Julien, 2017. "Using Split Samples to Improve Inference on Causal Effects," Political Analysis, Cambridge University Press, vol. 25(4), pages 465-482, October.
    59. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    60. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    61. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    62. Severnini, Edson R., 2014. "The Power of Hydroelectric Dams: Agglomeration Spillovers," IZA Discussion Papers 8082, Institute of Labor Economics (IZA).
    63. Erin O Sills & Diego Herrera & A Justin Kirkpatrick & Amintas Brandão Jr. & Rebecca Dickson & Simon Hall & Subhrendu Pattanayak & David Shoch & Mariana Vedoveto & Luisa Young & Alexander Pfaff, 2015. "Estimating the Impacts of Local Policy Innovation: The Synthetic Control Method Applied to Tropical Deforestation," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-15, July.
    64. Jason M. Lindo & Analisa Packham, 2017. "How Much Can Expanding Access to Long-Acting Reversible Contraceptives Reduce Teen Birth Rates?," American Economic Journal: Economic Policy, American Economic Association, vol. 9(3), pages 348-376, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roel Dom, 2017. "Semi-Autonomous Revenue Authorities in Sub-Saharan Africa: Silver Bullet or White Elephant," Discussion Papers 2017-01, University of Nottingham, CREDIT.
    2. Breinlich, Holger & Leromain, Elsa & Novy, Dennis & Sampson, Thomas, 2020. "Voting with their money: Brexit and outward investment by UK firms," European Economic Review, Elsevier, vol. 124(C).
    3. Dave, Dhaval M. & Friedson, Andrew I. & Matsuzawa, Kyutaro & McNichols, Drew & Sabia, Joseph J., 2020. "Did the Wisconsin Supreme Court Restart a COVID-19 Epidemic? Evidence from a Natural Experiment," IZA Discussion Papers 13314, Institute of Labor Economics (IZA).
    4. Jeffrey Clemens & Michael R. Strain, 2020. "Public Policy and Participation in Political Interest Groups: An Analysis of Minimum Wages, Labor Unions, and Effective Advocacy," NBER Working Papers 27902, National Bureau of Economic Research, Inc.
    5. Acel Jardón & Onno Kuik & Richard S.J. Tol, 2020. "Causal Effects of PetroCaribe on Sustainable Development: A Synthetic Control Analysis," Manchester School, University of Manchester, vol. 88(1), pages 156-210, January.
    6. Felipe Filgueiras, Elias Cavalcante-Filho, Rodrigo de Losso, José Roberto Savoia, 2019. "Law Change in a Regulated Sector Impacts Other Regulated Sectors: Evidence from Brazil," Working Papers, Department of Economics 2019_27, University of São Paulo (FEA-USP).
    7. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    8. Carvalho, Carlos & Masini, Ricardo & Medeiros, Marcelo C., 2018. "ArCo: An artificial counterfactual approach for high-dimensional panel time-series data," Journal of Econometrics, Elsevier, vol. 207(2), pages 352-380.
    9. Jason Poulos & Shuxi Zeng, 2017. "RNN-based counterfactual prediction, with an application to homestead policy and public schooling," Papers 1712.03553, arXiv.org, revised Sep 2020.
    10. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    11. Nathan, Max, 2019. "Does Light Touch Cluster Policy Work? Evaluating the Tech City Programme," IZA Discussion Papers 12631, Institute of Labor Economics (IZA).
    12. Ferman, Bruno & Pinto, Cristine, 2017. "Placebo Tests for Synthetic Controls," MPRA Paper 78079, University Library of Munich, Germany.
    13. Gabriel, Ricardo Duque & Pessoa, Ana Sofia, 2020. "Adopting the Euro: a synthetic control approach," MPRA Paper 99391, University Library of Munich, Germany.
    14. Ersoy, Fulya Y., 2020. "The effects of the great recession on college majors," Economics of Education Review, Elsevier, vol. 77(C).
    15. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.
    16. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    17. Bautista, María Angélica & González, Felipe & Martínez, Luis & Muñoz, Pablo & Prem, Mounu, 2020. "Chile’s Missing Students: Dictatorship, Higher Education and Social Mobility," Working papers 42, Red Investigadores de Economía.
    18. Kim, Hyejin & Lee, Jungmin, 2019. "Can employment subsidies save jobs? Evidence from a shipbuilding city in South Korea," Labour Economics, Elsevier, vol. 61(C).
    19. Martin Becker & Stefan Klößner & Gregor Pfeifer, 2018. "Cross-Validating Synthetic Controls," Economics Bulletin, AccessEcon, vol. 38(1), pages 603-609.
    20. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    21. Muhammad Jehangir Amjad & Devavrat Shah & Dennis Shen, 2017. "Robust Synthetic Control," Papers 1711.06940, arXiv.org.
    22. Michael Keller, 2020. "Oil Revenues vs Domestic Taxation: Deeper insights into the crowding-out effect," Working Paper Series 0120, Department of Economics, University of Sussex Business School.
    23. Bruno Ferman & Cristine Pinto, 2019. "Synthetic Controls with Imperfect Pre-Treatment Fit," Papers 1911.08521, arXiv.org.
    24. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fgv:eesptd:420. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Núcleo de Computação da FGV EPGE). General contact details of provider: http://edirc.repec.org/data/eegvfbr.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.