IDEAS home Printed from https://ideas.repec.org/a/wly/jpamgt/v26y2007i3p479-506.html
   My bibliography  Save this article

A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance program

Author

Listed:
  • Gary King

    (Harvard University)

  • Emmanuela Gakidou

    (Harvard University)

  • Nirmala Ravishankar

    (Harvard University)

  • Ryan T. Moore

    (Harvard University)

  • Jason Lakin

    (Harvard University)

  • Manett Vargas

    (National Commission for Social Protection in Health, Ministry of Health, Mexico)

  • Martha María Téllez-Rojo

    (Instituto Nacional de Salud Pública (National Institute of Public Health), Mexico)

  • Juan Eugenio Hernández Ávila

    (Instituto Nacional de Salud Pública (National Institute of Public Health), Mexico)

  • Mauricio Hernández Ávila

    (Undersecretary for Prevention and Health Promotion, Secretaría de Salud (Ministry of Health), Mexico)

  • Héctor Hernández Llamas

    (Conestadistica)

Abstract

We develop an approach to conducting large-scale randomized public policy experiments intended to be more robust to the political interventions that have ruined some or all parts of many similar previous efforts. Our proposed design is insulated from selection bias in some circumstances even if we lose observations; our inferences can still be unbiased even if politics disrupts any two of the three steps in our analytical procedures; and other empirical checks are available to validate the overall design. We illustrate with a design and empirical validation of an evaluation of the Mexican Seguro Popular de Salud (Universal Health Insurance) program we are conducting. Seguro Popular, which is intended to grow to provide medical care, drugs, preventative services, and financial health protection to the 50 million Mexicans without health insurance, is one of the largest health reforms of any country in the last two decades. The evaluation is also large scale, constituting one of the largest policy experiments to date and what may be the largest randomized health policy experiment ever. © 2007 by the Association for Public Policy Analysis and Management

Suggested Citation

  • Gary King & Emmanuela Gakidou & Nirmala Ravishankar & Ryan T. Moore & Jason Lakin & Manett Vargas & Martha María Téllez-Rojo & Juan Eugenio Hernández Ávila & Mauricio Hernández Ávila & Héctor Hernánde, 2007. "A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance program," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(3), pages 479-506.
  • Handle: RePEc:wly:jpamgt:v:26:y:2007:i:3:p:479-506
    DOI: 10.1002/pam.20279
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/pam.20279
    File Function: Link to full text; subscription required
    Download Restriction: no

    References listed on IDEAS

    as
    1. Nickerson, David W., 2005. "Scalable Protocols Offer Efficient Design for Field Experiments," Political Analysis, Cambridge University Press, vol. 13(03), pages 233-252, June.
    2. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    3. Glenn W. Harrison & John A. List, 2004. "Field Experiments," Journal of Economic Literature, American Economic Association, vol. 42(4), pages 1009-1055, December.
    4. Steven Glazerman & Daniel Mayer & Paul Decker, 2006. "Alternative routes to teaching: The impacts of Teach for America on student achievement and other outcomes," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(1), pages 75-96.
    5. Michael J. Camasso & Radha Jagannathan & Carol Harvey & Mark Killingsworth, 2003. "The use of client surveys to gauge the threat of contamination in welfare reform experiments," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 22(2), pages 207-223.
    6. Thomas S. Dee & Benjamin J. Keys, 2004. "Does merit pay reward good teachers? Evidence from a randomized experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 23(3), pages 471-488.
    7. Harry J. Holzer & John M. Quigley & Steven Raphael, 2003. "Public transit and the spatial distribution of minority employment: Evidence from a natural experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 22(3), pages 415-441.
    8. Elizabeth Ty Wilde & Robinson Hollister, 2007. "How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(3), pages 455-477.
    9. David H. Greenberg & Charles Michalopoulos & Philip K. Robin, 2006. "Do experimental and nonexperimental evaluations give different answers about the effectiveness of government-funded training programs?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(3), pages 523-552.
    10. Ho, Daniel & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2011. "MatchIt: Nonparametric Preprocessing for Parametric Causal Inference," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i08).
    11. repec:cup:apsrev:v:95:y:2001:i:01:p:49-69_00 is not listed on IDEAS
    12. Barnard J. & Frangakis C.E. & Hill J.L. & Rubin D.B., 2003. "Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City," Journal of the American Statistical Association, American Statistical Association, vol. 98, pages 299-323, January.
    13. William G. Howell, 2004. "Dynamic selection effects in means-tested, urban school voucher programs," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 23(2), pages 225-250.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Azuara, Oliver, 2011. "Effect of universal health coverage on marriage, cohabitation and labor force participation," MPRA Paper 35074, University Library of Munich, Germany.
    2. Sun, Xiaojie & Liu, Xiaoyun & Sun, Qiang & Yip, Winnie & Wagstaff, Adam & Meng, Qingyue, 2014. "The impact of a pay-for-performance scheme on prescription quality in rural China : an impact evaluation," Policy Research Working Paper Series 6892, The World Bank.
    3. Rodrigo Barros, 2008. "Wealthier But Not Much Healthier: Effects of a Health Insurance Program for the Poor in Mexico," Discussion Papers 09-002, Stanford Institute for Economic Policy Research.
    4. Beath, Andrew & Christia, Fotini & Enikolopov, Ruben, 2017. "Direct democracy and resource allocation: Experimental evidence from Afghanistan," Journal of Development Economics, Elsevier, vol. 124(C), pages 199-213.
    5. Pfutze, Tobias, 2014. "The Effects of Mexico’s Seguro Popular Health Insurance on Infant Mortality: An Estimation with Selection on the Outcome Variable," World Development, Elsevier, vol. 59(C), pages 475-486.
    6. Spenkch, Jörg L., 2011. "Adverse selection and moral hazard among the poor: evidence from a randomized experiment," MPRA Paper 31443, University Library of Munich, Germany.
    7. Groh, Matthew & McKenzie, David, 2016. "Macroinsurance for microenterprises: A randomized experiment in post-revolution Egypt," Journal of Development Economics, Elsevier, vol. 118(C), pages 13-25.
    8. Omar Galárraga & Sandra Sosa-Rubí & Aarón Salinas-Rodríguez & Sergio Sesma-Vázquez, 2010. "Health insurance for the poor: impact on catastrophic and out-of-pocket health expenditures in Mexico," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 11(5), pages 437-447, October.
    9. Azuara, Oliver & Marinescu, Ioana, 2011. "Informality and the expansion of social protection programs," MPRA Paper 35073, University Library of Munich, Germany.
    10. Zhang, Xuanchuan & Chen, Li-Wu & Mueller, Keith & Yu, Qiao & Liu, Jiapeng & Lin, Ge, 2011. "Tracking the effectiveness of health care reform in China: A case study of community health centers in a district of Beijing," Health Policy, Elsevier, vol. 100(2), pages 181-188.
    11. Rebecca L. Thornton & Laurel E. Hatt & Erica M. Field & Mursaleena Islam & Freddy Solís Diaz & Martha Azucena González, 2010. "Social security health insurance for the informal sector in Nicaragua: a randomized evaluation," Health Economics, John Wiley & Sons, Ltd., vol. 19(S1), pages 181-206, September.
    12. Wagstaff, Adam & Nguyen, Ha Thi Hong & Dao, Huyen & Balesd, Sarah, 2014. "Encouraging health insurance for the informal sector : a cluster randomized trial," Policy Research Working Paper Series 6910, The World Bank.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:jpamgt:v:26:y:2007:i:3:p:479-506. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing) or (Christopher F. Baum). General contact details of provider: http://www3.interscience.wiley.com/journal/34787/home .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.