IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/65074.html
   My bibliography  Save this paper

Examination of the synthetic control method for evaluating health policies with multiple treated units

Author

Listed:
  • Kreif, Noémi
  • Grieve, Richard
  • Hangartner, Dominik
  • Turner, Alex James
  • Nikolova, Silviya
  • Sutton, Matt

Abstract

This paper examines the synthetic control method in contrast to commonly used difference-in-differences (DiD) estimation, in the context of a re-evaluation of a pay-for-performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re-weighting the control group so that it has similar pre-intervention characteristics to the treated group. We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re-analyse a recent study evaluating the effects of a hospital P4P scheme on risk-adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non-incentivised conditions. This result was robust to alternative specifications of the synthetic control method

Suggested Citation

  • Kreif, Noémi & Grieve, Richard & Hangartner, Dominik & Turner, Alex James & Nikolova, Silviya & Sutton, Matt, 2016. "Examination of the synthetic control method for evaluating health policies with multiple treated units," LSE Research Online Documents on Economics 65074, London School of Economics and Political Science, LSE Library.
  • Handle: RePEc:ehl:lserod:65074
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/65074/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    2. Échevin, Damien & Fortin, Bernard, 2014. "Physician payment mechanisms, hospital length of stay and risk of readmission: Evidence from a natural experiment," Journal of Health Economics, Elsevier, vol. 36(C), pages 112-124.
    3. Currie, Janet & Stabile, Mark & Jones, Lauren, 2014. "Do stimulant medications improve educational and behavioral outcomes for children with ADHD?," Journal of Health Economics, Elsevier, vol. 37(C), pages 58-69.
    4. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    5. Brewer Mike & Crossley Thomas F. & Joyce Robert, 2018. "Inference with Difference-in-Differences Revisited," Journal of Econometric Methods, De Gruyter, vol. 7(1), pages 1-16, January.
    6. Ozkan Eren & I. Serkan Ozbeklik, 2011. "Right-to-Work Laws and State-Level Economic Outcomes: Evidence from the Case Studies of Idaho and Oklahoma Using Synthetic Control Method," Working Papers 1101 Classification-JEL J, University of Nevada, Las Vegas , Department of Economics.
    7. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    8. David Dranove & Ginger Zhe Jin, 2010. "Quality Disclosure and Certification: Theory and Practice," Journal of Economic Literature, American Economic Association, vol. 48(4), pages 935-963, December.
    9. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    10. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    11. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    12. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(2), pages 238-249, April.
    13. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    14. Charles J. Courtemanche & Daniela Zapata, 2014. "Does Universal Coverage Improve Health? The Massachusetts Experience," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(1), pages 36-69, January.
    15. Frank Eijkenaar, 2013. "Key issues in the design of pay for performance programs," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 14(1), pages 117-131, February.
    16. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, Oxford University Press, vol. 119(1), pages 249-275.
    17. Hugh Gravelle & Matt Sutton & Ada Ma, 2010. "Doctor Behaviour under a Pay for Performance Contract: Treating, Cheating and Case Finding?," Economic Journal, Royal Economic Society, vol. 120(542), pages 129-156, February.
    18. Coffman, Makena & Noy, Ilan, 2012. "Hurricane Iniki: measuring the long-term economic impact of a natural disaster using synthetic control," Environment and Development Economics, Cambridge University Press, vol. 17(2), pages 187-205, April.
    19. Rachel Meacock & Søren Rud Kristensen & Matt Sutton, 2014. "The Cost‐Effectiveness Of Using Financial Incentives To Improve Provider Quality: A Framework And Application," Health Economics, John Wiley & Sons, Ltd., vol. 23(1), pages 1-13, January.
    20. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    21. Martin Emmert & Frank Eijkenaar & Heike Kemter & Adelheid Esslinger & Oliver Schöffski, 2012. "Economic evaluation of pay-for-performance in health care: a systematic review," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 13(6), pages 755-767, December.
    22. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," Review of Economic Studies, Oxford University Press, vol. 72(1), pages 1-19.
    23. Roland G. Fryer, 2011. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," NBER Working Papers 16850, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    2. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Jul 2022.
    3. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    4. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research.
    5. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    6. Dietrichson, Jens & Ellegård, Lina Maria, 2015. "Assist or desist? Conditional bailouts and fiscal discipline in local governments," European Journal of Political Economy, Elsevier, vol. 38(C), pages 153-168.
    7. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    8. Erlend E. Bø & Joel Slemrod & Thor O. Thoresen, 2015. "Taxes on the Internet: Deterrence Effects of Public Disclosure," American Economic Journal: Economic Policy, American Economic Association, vol. 7(1), pages 36-62, February.
    9. Olper, Alessandro & Curzi, Daniele & Swinnen, Johan, 2018. "Trade liberalization and child mortality: A Synthetic Control Method," World Development, Elsevier, vol. 110(C), pages 394-410.
    10. Klößner, Stefan & Pfeifer, Gregor, 2015. "Synthesizing Cash for Clunkers: Stabilizing the Car Market, Hurting the Environment," VfS Annual Conference 2015 (Muenster): Economic Development - Theory and Policy 113207, Verein für Socialpolitik / German Economic Association.
    11. Myoung-jae Lee & Yasuyuki Sawada, 2020. "Review on Difference in Differences," Korean Economic Review, Korean Economic Association, vol. 36, pages 135-173.
    12. Cruz A. Echevarría & Javier García-Enríquez, 2020. "The economic cost of the Arab Spring: the case of the Egyptian revolution," Empirical Economics, Springer, vol. 59(3), pages 1453-1477, September.
    13. Rok Spruk & Mitja Kovac, 2020. "Does a ban on trans fats improve public health: synthetic control evidence from Denmark," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 156(1), pages 1-32, December.
    14. Barseghyan, Gayane, 2019. "Sanctions and counter-sanctions : What did they do?," BOFIT Discussion Papers 24/2019, Bank of Finland, Institute for Economies in Transition.
    15. Nauro Campos & Fabrizio Coricelli & Luigi Moretti,, 2020. "Choosing Institutional over Economic Integration: Are There Growth Effects?," Working Papers hal-03028143, HAL.
    16. Victor Chernozhukov & Kaspar Wüthrich & Yinchu Zhu, 2021. "An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1849-1864, October.
    17. Marchesi, Silvia & Masi, Tania, 2021. "Life after default. Private and official deals," Journal of International Money and Finance, Elsevier, vol. 113(C).
    18. Chung, Sunghoon & Lee, Joonhyung & Osang, Thomas, 2016. "Did China tire safeguard save U.S. workers?," European Economic Review, Elsevier, vol. 85(C), pages 22-38.
    19. Bibek Adhikari & James Alm, 2016. "Evaluating the Economic Effects of Flat Tax Reforms Using Synthetic Control Methods," Southern Economic Journal, John Wiley & Sons, vol. 83(2), pages 437-463, October.
    20. Irene Botosaru & Bruno Ferman, 2019. "On the role of covariates in the synthetic control method," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 117-130.

    More about this item

    Keywords

    synthetic control method; difference-in-differences; policy evaluation; pay-for-performance; SRF-2013-06-016; MR/L012332/1;
    All these keywords.

    JEL classification:

    • N0 - Economic History - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:65074. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.