IDEAS home Printed from https://ideas.repec.org/a/bpj/causin/v8y2020i1p209-228n10.html

When is a Match Sufficient? A Score-based Balance Metric for the Synthetic Control Method

Author

Listed:
  • Parast Layla

    (RAND Corporation, 1776Main Street, Santa Monica, CAU.S.A)

  • Hunt Priscillia

    (RAND Corporation, 1776Main Street, Santa Monica, CAU.S.A)

  • Griffin Beth Ann

    (RAND Corporation, 1200 S Hayes St, Arlington, VA22202U.S.A)

  • Powell David

    (RAND Corporation, 1200 S Hayes St, Arlington, VA22202U.S.A)

Abstract

In some applications, researchers using the synthetic control method (SCM) to evaluate the effect of a policy may struggle to determine whether they have identified a “good match” between the control group and treated group. In this paper, we demonstrate the utility of the mean and maximum Absolute Standardized Mean Difference (ASMD) as a test of balance between a synthetic control unit and treated unit, and provide guidance on what constitutes a poor fit when using a synthetic control. We explore and compare other potential metrics using a simulation study. We provide an application of our proposed balance metric to the 2013 Los Angeles (LA) Firearm Study [9]. Using Uniform Crime Report data, we apply the SCM to obtain a counterfactual for the LA firearm-related crime rate based on a weighted combination of control units in a donor pool of cities. We use this counterfactual to estimate the effect of the LA Firearm Study intervention and explore the impact of changing the donor pool and pre-intervention duration period on resulting matches and estimated effects. We demonstrate how decision-making about the quality of a synthetic control can be improved by using ASMD. The mean and max ASMD clearly differentiate between poor matches and good matches. Researchers need better guidance on what is a meaningful imbalance between synthetic control and treated groups. In addition to the use of gap plots, the proposed balance metric can provide an objective way of determining fit.

Suggested Citation

  • Parast Layla & Hunt Priscillia & Griffin Beth Ann & Powell David, 2020. "When is a Match Sufficient? A Score-based Balance Metric for the Synthetic Control Method," Journal of Causal Inference, De Gruyter, vol. 8(1), pages 209-228, January.
    • Handle: RePEc:bpj:causin:v:8:y:2020:i:1:p:209-228:n:10
    DOI: 10.1515/jci-2020-0013
    as

    Download full text from publisher

    File URL: https://doi.org/10.1515/jci-2020-0013
    Download Restriction: no
    File URL: https://libkey.io/10.1515/jci-2020-0013?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. José R. Zubizarreta, 2015. "Stable Weights that Balance Covariates for Estimation With Incomplete Outcome Data," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 910-922, September.
    2. Dmitry Arkhangelsky & Susan Athey & David A. Hirshberg & Guido W. Imbens & Stefan Wager, 2021. "Synthetic Difference-in-Differences," American Economic Review, American Economic Association, vol. 111(12), pages 4088-4118, December.
    3. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    4. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2011. "Synth: An R Package for Synthetic Control Methods in Comparative Case Studies," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i13).
    5. Paolo Pinotti, 2015. "The Economic Costs of Organised Crime: Evidence from Southern Italy," Economic Journal, Royal Economic Society, vol. 125(586), pages 203-232, August.
    6. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    7. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    8. Michael W. Robbins & Jessica Saunders & Beau Kilmer, 2017. "A Framework for Synthetic Control Methods With High-Dimensional, Micro-Level Data: Evaluating a Neighborhood-Specific Crime Intervention," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(517), pages 109-126, January.
    9. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    10. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    11. Nikolay Doudchenko & Guido W. Imbens, 2016. "Balancing, Regression, Difference-In-Differences and Synthetic Control Methods: A Synthesis," NBER Working Papers 22791, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Di Novi; C.; & Salari; P.;, 2026. "Shaping Soft Drinks: Sugar Taxes, Consumption, and Reformulation in Europe," Health, Econometrics and Data Group (HEDG) Working Papers 26/07, HEDG, c/o Department of Economics, University of York.
    2. Patra, Kamalika & Neuteboom, Nora & Kapetanios, George & Ventouri, Alexia, 2025. "Reducing labour market flexibility: A causal inference study on reform in The Netherlands," Economics Letters, Elsevier, vol. 247(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dennis Shen & Peng Ding & Jasjeet Sekhon & Bin Yu, 2022. "Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data," Papers 2207.14481, arXiv.org, revised Oct 2022.
    2. Taehyeon Koo & Zijian Guo, 2025. "Distributionally Robust Synthetic Control: Ensuring Robustness Against Highly Correlated Controls and Weight Shifts," Papers 2511.02632, arXiv.org, revised Jan 2026.
    3. Yi‐Ting Chen, 2020. "A distributional synthetic control method for policy evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 505-525, August.
    4. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    5. Pekka Malo & Juha Eskelinen & Xun Zhou & Timo Kuosmanen, 2024. "Computing Synthetic Controls Using Bilevel Optimization," Computational Economics, Springer;Society for Computational Economics, vol. 64(2), pages 1113-1136, August.
    6. Garoupa, Nuno & Spruk, Rok, 2025. "Populist constitutional backsliding and judicial independence: Evidence from Türkiye," International Review of Law and Economics, Elsevier, vol. 84(C).
    7. Cho, Sang-Wook (Stanley) & Wong, Sally, 2025. "Monetary regimes and regional economies: A counterfactual perspective from two euro opt-outs," Journal of Macroeconomics, Elsevier, vol. 85(C).
    8. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    9. Billy, Alexander & Packard, Michael, 2022. "Crime and the Mariel Boatlift," International Review of Law and Economics, Elsevier, vol. 72(C).
    10. Roy Cerqueti & Raffaella Coppier & Alessandro Girardi & Marco Ventura, 2022. "The sooner the better: lives saved by the lockdown during the COVID-19 outbreak. The case of Italy," The Econometrics Journal, Royal Economic Society, vol. 25(1), pages 46-70.
    11. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    12. Alice Lépissier & Matto Mildenberger, 2021. "Unilateral climate policies can substantially reduce national carbon pollution," Climatic Change, Springer, vol. 166(3), pages 1-21, June.
    13. Kraynak,Daniel Christopher & Timilsina,Govinda R. & Alberini,Anna, 2024. "The Effect of Pricing Instruments on CO2 Emissions: Empirical Evidence from Australia," Policy Research Working Paper Series 10812, The World Bank.
    14. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2021. "Decoupling synthetic control methods to ensure stability, accuracy and meaningfulness," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 12(4), pages 549-584, December.
    15. Kim, Youngho, 2023. "Payments for Ecosystem Services Programs and Climate Change Adaptation in Agriculture," 2023 Annual Meeting, July 23-25, Washington D.C. 335971, Agricultural and Applied Economics Association.
    16. Youngho Kim, 2024. "Payments for Ecosystem Services Programs and Climate Change Adaptation in Agriculture," Economics Series Working Papers 1054, University of Oxford, Department of Economics.
    17. repec:ags:aaea22:335971 is not listed on IDEAS
    18. Samuel Verevis & Murat Üngör, 2021. "What has New Zealand gained from The FTA with China?: Two counterfactual analyses†," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(1), pages 20-50, February.
    19. Masahiro Kato & Akari Ohda, 2023. "Asymptotically Unbiased Synthetic Control Methods by Moment Matching," Papers 2307.11127, arXiv.org, revised Nov 2025.
    20. Nicolaj N. Mühlbach, 2020. "Tree-based Synthetic Control Methods: Consequences of moving the US Embassy," CREATES Research Papers 2020-04, Department of Economics and Business Economics, Aarhus University.
    21. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:causin:v:8:y:2020:i:1:p:209-228:n:10. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyterbrill.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.