IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/65074.html

Examination of the synthetic control method for evaluating health policies with multiple treated units

Author

Listed:
  • Kreif, Noémi
  • Grieve, Richard
  • Hangartner, Dominik
  • Turner, Alex James
  • Nikolova, Silviya
  • Sutton, Matt

Abstract

This paper examines the synthetic control method in contrast to commonly used difference-in-differences (DiD) estimation, in the context of a re-evaluation of a pay-for-performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re-weighting the control group so that it has similar pre-intervention characteristics to the treated group. We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re-analyse a recent study evaluating the effects of a hospital P4P scheme on risk-adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non-incentivised conditions. This result was robust to alternative specifications of the synthetic control method

Suggested Citation

  • Kreif, Noémi & Grieve, Richard & Hangartner, Dominik & Turner, Alex James & Nikolova, Silviya & Sutton, Matt, 2016. "Examination of the synthetic control method for evaluating health policies with multiple treated units," LSE Research Online Documents on Economics 65074, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:65074
    as

    Download full text from publisher

    File URL: https://researchonline.lse.ac.uk/id/eprint/65074/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Coffman, Makena & Noy, Ilan, 2012. "Hurricane Iniki: measuring the long-term economic impact of a natural disaster using synthetic control," Environment and Development Economics, Cambridge University Press, vol. 17(2), pages 187-205, April.
    2. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    3. Rachel Meacock & Søren Rud Kristensen & Matt Sutton, 2014. "The Cost‐Effectiveness Of Using Financial Incentives To Improve Provider Quality: A Framework And Application," Health Economics, John Wiley & Sons, Ltd., vol. 23(1), pages 1-13, January.
    4. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    5. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    6. Charles J. Courtemanche & Daniela Zapata, 2014. "Does Universal Coverage Improve Health? The Massachusetts Experience," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(1), pages 36-69, January.
    7. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    8. Susan Athey & Guido W. Imbens, 2006. "Identification and Inference in Nonlinear Difference-in-Differences Models," Econometrica, Econometric Society, vol. 74(2), pages 431-497, March.
    9. Martin Emmert & Frank Eijkenaar & Heike Kemter & Adelheid Esslinger & Oliver Schöffski, 2012. "Economic evaluation of pay-for-performance in health care: a systematic review," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 13(6), pages 755-767, December.
    10. Cameron, A. Colin & Gelbach, Jonah B. & Miller, Douglas L., 2011. "Robust Inference With Multiway Clustering," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 238-249.
    11. Frank Eijkenaar, 2013. "Key issues in the design of pay for performance programs," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 14(1), pages 117-131, February.
    12. Échevin, Damien & Fortin, Bernard, 2014. "Physician payment mechanisms, hospital length of stay and risk of readmission: Evidence from a natural experiment," Journal of Health Economics, Elsevier, vol. 36(C), pages 112-124.
    13. Sekhon, Jasjeet S., 2011. "Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching package for R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i07).
    14. Brewer Mike & Crossley Thomas F. & Joyce Robert, 2018. "Inference with Difference-in-Differences Revisited," Journal of Econometric Methods, De Gruyter, vol. 7(1), pages 1-16, January.
    15. Ozkan Eren & I. Serkan Ozbeklik, 2011. "Right-to-Work Laws and State-Level Economic Outcomes: Evidence from the Case Studies of Idaho and Oklahoma Using Synthetic Control Method," Working Papers 1101 Classification-JEL J, University of Nevada, Las Vegas , Department of Economics.
    16. Currie, Janet & Stabile, Mark & Jones, Lauren, 2014. "Do stimulant medications improve educational and behavioral outcomes for children with ADHD?," Journal of Health Economics, Elsevier, vol. 37(C), pages 58-69.
    17. Hugh Gravelle & Matt Sutton & Ada Ma, 2010. "Doctor Behaviour under a Pay for Performance Contract: Treating, Cheating and Case Finding?," Economic Journal, Royal Economic Society, vol. 120(542), pages 129-156, February.
    18. David Dranove & Ginger Zhe Jin, 2010. "Quality Disclosure and Certification: Theory and Practice," Journal of Economic Literature, American Economic Association, vol. 48(4), pages 935-963, December.
    19. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    20. Roland G. Fryer, 2011. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," NBER Working Papers 16850, National Bureau of Economic Research, Inc.
    21. Peter Hinrichs, 2012. "The Effects of Affirmative Action Bans on College Enrollment, Educational Attainment, and the Demographic Composition of Universities," The Review of Economics and Statistics, MIT Press, vol. 94(3), pages 712-722, August.
    22. Acemoglu, Daron & Johnson, Simon & Kermani, Amir & Kwak, James & Mitton, Todd, 2016. "The value of connections in turbulent times: Evidence from the United States," Journal of Financial Economics, Elsevier, vol. 121(2), pages 368-391.
    23. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    24. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    2. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    3. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    4. Kleis, Mischa & Moessinger, Marc-Daniel, 2016. "The long-run effect of fiscal consolidation on economic growth: Evidence from quantitative case studies," ZEW Discussion Papers 16-047, ZEW - Leibniz Centre for European Economic Research, revised 2016.
    5. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    6. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    7. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    8. Olper, Alessandro & Curzi, Daniele & Swinnen, Johan, 2018. "Trade liberalization and child mortality: A Synthetic Control Method," World Development, Elsevier, vol. 110(C), pages 394-410.
    9. Myoung-jae Lee & Yasuyuki Sawada, 2020. "Review on Difference in Differences," Korean Economic Review, Korean Economic Association, vol. 36, pages 135-173.
    10. Tendai Gwatidzo, 2025. "Analysing the impact of climate change on economic growth in the SADC region a synthetic control approach," Working Papers 11091, South African Reserve Bank.
    11. Ben Deaner & Hyejin Ku, 2024. "Causal Duration Analysis with Diff-in-Diff," Papers 2405.05220, arXiv.org.
    12. Cummins Joseph & Miller Douglas L. & Smith Brock & Simon David, 2024. "Matching on Noise: Finite Sample Bias in the Synthetic Control Estimator," Journal of Econometric Methods, De Gruyter, vol. 13(1), pages 67-95, January.
    13. Cruz A. Echevarría & Javier García-Enríquez, 2020. "The economic cost of the Arab Spring: the case of the Egyptian revolution," Empirical Economics, Springer, vol. 59(3), pages 1453-1477, September.
    14. Rok Spruk & Mitja Kovac, 2020. "Does a ban on trans fats improve public health: synthetic control evidence from Denmark," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 156(1), pages 1-32, December.
    15. Cruz A. Echevarría & Serhat Hasancebi & Javier García-Enríquez, 2022. "Economic Effects of Macao’s Integration with Mainland China: A Causal Inference Study," Journal of Economic Integration, Center for Economic Integration, Sejong University, vol. 37(2), pages 179-215.
    16. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    17. Chuku Chuku & Mustafa Yasin Yenice, 2021. "Working Paper 356 - Eurobonds, debt sustainability and macroeconomic performance in Africa: Synthetic controlled experiments," Working Paper Series 2482, African Development Bank.
    18. repec:osf:osfxxx:s8ayp_v1 is not listed on IDEAS
    19. Zhentao Shi & Jin Xi & Haitian Xie, 2025. "A Synthetic Business Cycle Approach to Counterfactual Analysis with Nonstationary Macroeconomic Data," Papers 2505.22388, arXiv.org.
    20. Barseghyan, Gayane, 2019. "Sanctions and counter-sanctions : What did they do?," BOFIT Discussion Papers 24/2019, Bank of Finland, Institute for Economies in Transition.
    21. Ryan Abman & Hisham Foad, 2022. "Border Walls and Crime: Evidence From the Secure Fence Act," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 48(2), pages 167-197, April.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • N0 - Economic History - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:65074. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.