IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i6p593-619.html
   My bibliography  Save this article

Using Synthetic Controls to Evaluate the Effect of Unique Interventions: The Case of Say Yes to Education

Author

Listed:
  • Robert Bifulco
  • Ross Rubenstein
  • Hosung Sohn

Abstract

Background: “Place-based†scholarships seek to improve student outcomes in urban school districts and promote urban revitalization in economically challenged cities. Say Yes to Education is a unique district-wide school reform effort adopted in Syracuse, NY, in 2008. It includes full-tuition scholarships for public and private universities, coupled with extensive wraparound support services in schools. Objectives: This study uses synthetic control methods to evaluate the effect of Say Yes on district enrollment and graduation rates. It also introduces the synthetic control method and provides guidance for its use in evaluating single-site interventions. Method: Combining school district-level data from the National Center for Education Statistics’ Common Core of Data and New York State School Report Cards, this article uses synthetic control methods to construct a synthetic comparison district to estimate counterfactual enrollment and graduation trends for Syracuse. Results: We find that Say Yes to Education was associated with enrollment increases in the Syracuse City School District, a district that had previously experienced decades of sustained enrollment declines. We do not find consistent evidence of changes in graduation rates following adoption of the program. Conclusions: Graduation rate analyses demonstrate that estimates of treatment effects can be sensitive to choices that the researcher has to make in applying synthetic control methods, particularly when pretreatment outcome measures appear to have considerable amounts of noise.

Suggested Citation

  • Robert Bifulco & Ross Rubenstein & Hosung Sohn, 2017. "Using Synthetic Controls to Evaluate the Effect of Unique Interventions: The Case of Say Yes to Education," Evaluation Review, , vol. 41(6), pages 593-619, December.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:6:p:593-619
    DOI: 10.1177/0193841X17742233
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X17742233
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X17742233?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mircea Trandafir, 2014. "The Effect of Same-Sex Marriage Laws on Different-Sex Marriage: Evidence From the Netherlands," Demography, Springer;Population Association of America (PAA), vol. 51(1), pages 317-340, February.
    2. LeGower, Michael & Walsh, Randall, 2017. "Promise scholarship programs as place-making policy: Evidence from school enrollment and housing prices," Journal of Urban Economics, Elsevier, vol. 101(C), pages 74-89.
    3. Brad J. Hershbein, 2013. "A Second Look at Enrollment Changes after the Kalamazoo Promise," Upjohn Working Papers 13-200, W.E. Upjohn Institute for Employment Research.
    4. Sarah Bohn & Magnus Lofstrom & Steven Raphael, 2014. "Did the 2007 Legal Arizona Workers Act Reduce the State's Unauthorized Immigrant Population?," The Review of Economics and Statistics, MIT Press, vol. 96(2), pages 258-269, May.
    5. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    6. Alberto Abadie & Javier Gardeazabal, 2003. "The Economic Costs of Conflict: A Case Study of the Basque Country," American Economic Review, American Economic Association, vol. 93(1), pages 113-132, March.
    7. Coricelli, Fabrizio & Campos, Nauro & Moretti, Luigi, 2014. "Economic Growth and Political Integration: Estimating the Benefits from Membership in the European Union Using the Synthetic Co," CEPR Discussion Papers 9968, C.E.P.R. Discussion Papers.
    8. Campos, Nauro F. & Coricelli, Fabrizio & Moretti, Luigi, 2014. "Economic Growth and Political Integration: Estimating the Benefits from Membership in the European Union Using the Synthetic Counterfactuals Method," IZA Discussion Papers 8162, Institute of Labor Economics (IZA).
    9. Andreas Billmeier & Tommaso Nannicini, 2013. "Assessing Economic Liberalization Episodes: A Synthetic Control Approach," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 983-1001, July.
    10. Timothy J. Bartik & Nathan Sotherland, 2015. "Migration and Housing Price Effects of Place-Based College Scholarships," Upjohn Working Papers 15-245, W.E. Upjohn Institute for Employment Research.
    11. Timothy J. Bartik & Brad J. Hershbein & Marta Lachowska, 2015. "The Effects of the Kalamazoo Promise Scholarship on College Enrollment, Persistence, and Completion," Upjohn Working Papers 15-228, W.E. Upjohn Institute for Employment Research.
    12. Timothy J. Bartik & Marta Lachowska, "undated". "The Kalamazoo Promise Scholarship," Upjohn Working Papers tjbml14, W.E. Upjohn Institute for Employment Research.
    13. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    14. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Guillaume Allaire Pouliot & Zhen Xie, 2022. "Degrees of Freedom and Information Criteria for the Synthetic Control Method," Papers 2207.02943, arXiv.org.
    2. Elaine W. Leigh & Manuel S. González Canché, 2021. "The College Promise in Communities: Do Place-based Scholarships Affect Residential Mobility Patterns?," Research in Higher Education, Springer;Association for Institutional Research, vol. 62(3), pages 259-308, May.
    3. Brett Parker, 2021. "Death Penalty Statutes and Murder Rates: Evidence From Synthetic Controls," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 18(3), pages 488-533, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dennis Essers & Stefaan Ide, 2017. "The IMF and precautionary lending : An empirical evaluation of the selectivity and effectiveness of the flexible credit line," Working Paper Research 323, National Bank of Belgium.
    2. Essers, Dennis & Ide, Stefaan, 2019. "The IMF and precautionary lending: An empirical evaluation of the selectivity and effectiveness of the Flexible Credit Line," Journal of International Money and Finance, Elsevier, vol. 92(C), pages 25-61.
    3. Erin O Sills & Diego Herrera & A Justin Kirkpatrick & Amintas Brandão Jr. & Rebecca Dickson & Simon Hall & Subhrendu Pattanayak & David Shoch & Mariana Vedoveto & Luisa Young & Alexander Pfaff, 2015. "Estimating the Impacts of Local Policy Innovation: The Synthetic Control Method Applied to Tropical Deforestation," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-15, July.
    4. Katarzyna Metelska-Szaniawska, 2016. "Reassessing the Economic Effects of Post-Socialist Constitutions Using the Synthetic Control Method," Working Papers 2016-18, Faculty of Economic Sciences, University of Warsaw.
    5. Barati Mehdi, 2019. "Punishment Severity and Crime: The Case of Arkansas," Review of Law & Economics, De Gruyter, vol. 15(1), pages 1-23, March.
    6. Daniel Albalate & Germà Bel & Ferran A. Mazaira-Font, 2020. "Ensuring Stability, Accuracy and Meaningfulness in Synthetic Control Methods: The Regularized SHAP-Distance Method," IREA Working Papers 202005, University of Barcelona, Research Institute of Applied Economics, revised Apr 2020.
    7. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    8. Maximiliano Marzetti & Rok Spruk, 2023. "Long-Term Economic Effects of Populist Legal Reforms: Evidence from Argentina," Comparative Economic Studies, Palgrave Macmillan;Association for Comparative Economic Studies, vol. 65(1), pages 60-95, March.
    9. Bibek Adhikari & Romain Duval & Bingjie Hu & Prakash Loungani, 2018. "Can Reform Waves Turn the Tide? Some Case Studies using the Synthetic Control Method," Open Economies Review, Springer, vol. 29(4), pages 879-910, September.
    10. Niklas Potrafke & Fabian Ruthardt & Kaspar Wuthrich, 2020. "Protectionism and economic growth: Causal evidence from the first era of globalization," Papers 2010.02378, arXiv.org, revised Mar 2022.
    11. Johanna Catherine Maclean & Brendan Saloner, 2018. "Substance Use Treatment Provider Behavior and Healthcare Reform: Evidence from Massachusetts," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 76-101, January.
    12. Ryan Abman & Hisham Foad, 2022. "Border Walls and Crime: Evidence From the Secure Fence Act," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 48(2), pages 167-197, April.
    13. Robbert Maseland & Rok Spruk, 2023. "The benefits of US statehood: an analysis of the growth effects of joining the USA," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(1), pages 49-89, January.
    14. Nauro Campos & Fabrizio Coricelli & Luigi Moretti, 2015. "Norwegian Rhapsody? The Political Economy Benefits of Regional Integration," Working Papers halshs-01267252, HAL.
    15. Makram El-Shagi & Axel Lindner & Gregor von Schweinitz, 2016. "Real Effective Exchange Rate Misalignment in the Euro Area: A Counterfactual Analysis," Review of International Economics, Wiley Blackwell, vol. 24(1), pages 37-66, February.
    16. Giulio Grossi & Marco Mariani & Alessandra Mattei & Patrizia Lattarulo & Ozge Oner, 2020. "Direct and spillover effects of a new tramway line on the commercial vitality of peripheral streets. A synthetic-control approach," Papers 2004.05027, arXiv.org, revised Nov 2023.
    17. David Gilchrist & Thomas Emery & Nuno Garoupa & Rok Spruk, 2023. "Synthetic Control Method: A tool for comparative case studies in economic history," Journal of Economic Surveys, Wiley Blackwell, vol. 37(2), pages 409-445, April.
    18. Maïmouna Diakite & Jean-François Brun & Souleymane Diarra & Nasser Ary Tanimoune, 2017. "The effects of tax coordination on the tax revenue mobilization in West African Economic and Monetary Union (WAEMU)," CERDI Working papers halshs-01535104, HAL.
    19. Kaul, Ashok & Klößner, Stefan & Pfeifer, Gregor & Schieler, Manuel, 2015. "Synthetic Control Methods: Never Use All Pre-Intervention Outcomes Together With Covariates," MPRA Paper 83790, University Library of Munich, Germany.
    20. Pier Basaglia & Sophie Behr & Moritz A. Drupp, 2023. "De-Fueling Externalities: How Tax Salience and Fuel Substitution Mediate Climate and Health Benefits," Discussion Papers of DIW Berlin 2041, DIW Berlin, German Institute for Economic Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:6:p:593-619. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.