IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/78213.html
   My bibliography  Save this paper

Cherry Picking with Synthetic Controls

Author

Listed:
  • Ferman, Bruno
  • Pinto, Cristine
  • Possebom, Vitor

Abstract

We show that a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities in SC applications. This undermines one of the potential advantages of the method, which is providing a transparent way of choosing comparison units and, therefore, being less susceptible to specification searching than alternative methods. To address this problem, we provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and our recommendations in two empirical applications.

Suggested Citation

  • Ferman, Bruno & Pinto, Cristine & Possebom, Vitor, 2017. "Cherry Picking with Synthetic Controls," MPRA Paper 78213, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:78213
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/78213/1/MPRA_paper_78213.pdf
    File Function: original version
    Download Restriction: no

    File URL: https://mpra.ub.uni-muenchen.de/80970/1/MPRA_paper_80970.pdf
    File Function: revised version
    Download Restriction: no

    File URL: https://mpra.ub.uni-muenchen.de/85138/1/MPRA_paper_85138.pdf
    File Function: revised version
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Isabela Ferreira Duarte & João Manoel Pinho de Mello & Vinicius Nascimento Carrasco, 2014. "A Década Perdida: 2003 – 2012," Textos para discussão 626, Department of Economics PUC-Rio (Brazil).
    2. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    3. Laurent Gobillon & Thierry Magnac, 2016. "Regional Policy Evaluation: Interactive Fixed Effects and Synthetic Controls," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 535-551, July.
    4. Gabriela Calderón, 2014. "The Effects of Child Care Provision in Mexico," Working Papers 2014-07, Banco de México.
    5. José G. Montalvo, 2011. "Voting after the Bombings: A Natural Experiment on the Effect of Terrorist Attacks on Democratic Elections," The Review of Economics and Statistics, MIT Press, vol. 93(4), pages 1146-1154, November.
    6. Vincenzo Bove & Leandro Eliay & Ron P Smith, 2014. "The relationship between panel and synthetic control estimators of the effect of civil war," BCAM Working Papers 1406, Birkbeck Centre for Applied Macroeconomics.
    7. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    8. Andreas Billmeier & Tommaso Nannicini, 2009. "Trade Openness and Growth: Pursuing Empirical Glasnost," IMF Staff Papers, Palgrave Macmillan, vol. 56(3), pages 447-475, August.
    9. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    10. Ando, Michihito, 2015. "Dreams of urbanization: Quantitative case studies on the local impacts of nuclear power facilities using the synthetic control method," Journal of Urban Economics, Elsevier, vol. 85(C), pages 68-85.
    11. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    12. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    13. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, Southern Economic Association, vol. 83(2), pages 437-463, October.
    14. Eduardo Cavallo & Sebastian Galiani & Ilan Noy & Juan Pantano, 2013. "Catastrophic Natural Disasters and Economic Growth," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1549-1561, December.
    15. repec:wly:hlthec:v:25:y:2016:i:12:p:1514-1528 is not listed on IDEAS
    16. Marcos Sanso‐Navarro, 2011. "The Effects on American Foreign Direct Investment in the United Kingdom from Not Adopting the Euro," Journal of Common Market Studies, Wiley Blackwell, vol. 49(2), pages 463-483, March.
    17. Noémi Kreif & Richard Grieve & Dominik Hangartner & Alex James Turner & Silviya Nikolova & Matt Sutton, 2016. "Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units," Health Economics, John Wiley & Sons, Ltd., vol. 25(12), pages 1514-1528, December.
    18. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    19. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    20. Jinjarak, Yothin & Noy, Ilan & Zheng, Huanhuan, 2013. "Capital controls in Brazil – Stemming a tide with a signal?," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 2938-2952.
    21. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    22. Henrik Jacobsen Kleven & Camille Landais & Emmanuel Saez, 2013. "Taxation and International Migration of Superstars: Evidence from the European Football Market," American Economic Review, American Economic Association, vol. 103(5), pages 1892-1924, August.
    23. Barone, Guglielmo & Mocetti, Sauro, 2014. "Natural disasters, growth and institutions: A tale of two earthquakes," Journal of Urban Economics, Elsevier, vol. 84(C), pages 52-66.
    24. Amr Hosny, 2012. "Algeria’s Trade with GAFTA Countries: A Synthetic Control Approach," Transition Studies Review, Springer;Central Eastern European University Network (CEEUN), vol. 19(1), pages 35-42, September.
    25. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    26. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2011. "Synth: An R Package for Synthetic Control Methods in Comparative Case Studies," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i13).
    27. Shimeng Liu, 2015. "Spillovers from Universities: Evidence from the Land-Grant Program," Working Paper 9410, USC Lusk Center for Real Estate.
    28. William duPont IV & Ilan Noy, 2015. "What Happened to Kobe? A Reassessment of the Impact of the 1995 Earthquake in Japan," Economic Development and Cultural Change, University of Chicago Press, vol. 63(4), pages 777-812.
    29. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    30. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    31. Bauhoff, Sebastian, 2014. "The effect of school district nutrition policies on dietary intake and overweight: A synthetic control approach," Economics & Human Biology, Elsevier, vol. 12(C), pages 45-55.
    32. Liu, Shimeng, 2015. "Spillovers from universities: Evidence from the land-grant program," Journal of Urban Economics, Elsevier, vol. 87(C), pages 25-41.
    33. Ferman, Bruno & Pinto, Cristine Campos de Xavier, 2016. "Revisiting the synthetic control estimator," Textos para discussão 421, FGV/EESP - Escola de Economia de São Paulo, Getulio Vargas Foundation (Brazil).
    34. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    35. Garret S. Christensen & Edward Miguel, 2016. "Transparency, Reproducibility, and the Credibility of Economics Research," NBER Working Papers 22989, National Bureau of Economic Research, Inc.
    36. Mideksa, Torben K., 2013. "The economic impact of natural resources," Journal of Environmental Economics and Management, Elsevier, vol. 65(2), pages 277-289.
    37. Smith, Brock, 2015. "The resource curse exorcised: Evidence from a panel of countries," Journal of Development Economics, Elsevier, vol. 116(C), pages 57-73.
    38. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    39. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    40. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    41. Severnini, Edson R., 2014. "The Power of Hydroelectric Dams: Agglomeration Spillovers," IZA Discussion Papers 8082, Institute for the Study of Labor (IZA).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Acel Jardón & Onno Kuik & Richard S.J. Tol, 2018. "Causal effects of PetroCaribe on sustainable development: a synthetic control analysis," Working Paper Series 0918, Department of Economics, University of Sussex.
    2. Ferman, Bruno & Pinto, Cristine, 2016. "Revisiting the Synthetic Control Estimator," MPRA Paper 73982, University Library of Munich, Germany.
    3. Botosaru, Irene & Ferman, Bruno, 2017. "On the Role of Covariates in the Synthetic Control Method," MPRA Paper 80796, University Library of Munich, Germany.
    4. repec:ebl:ecbull:eb-17-00821 is not listed on IDEAS
    5. Martin Becker & Stefan Klößner & Gregor Pfeifer, 2018. "Cross-Validating Synthetic Controls," Economics Bulletin, AccessEcon, vol. 38(1), pages 603-609.
    6. Muhammad Jehangir Amjad & Devavrat Shah & Dennis Shen, 2017. "Robust Synthetic Control," Papers 1711.06940, arXiv.org.
    7. Roel Dom, 2017. "Semi-Autonomous Revenue Authorities in Sub-Saharan Africa: Silver Bullet or White Elephant," Discussion Papers 2017-01, University of Nottingham, CREDIT.
    8. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    9. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Zentrum für Europäische Wirtschaftsforschung / Center for European Economic Research.

    More about this item

    Keywords

    inference; synthetic control; p-hacking; specification searching;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:78213. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Joachim Winter). General contact details of provider: http://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.