IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/16016.html
   My bibliography  Save this paper

Program Evaluation and Research Designs

Author

Listed:
  • John DiNardo
  • David S. Lee

Abstract

This chapter provides a selective review of some contemporary approaches to program evaluation. One motivation for our review is the recent emergence and increasing use of a particular kind of "program" in applied microeconomic research, the so-called Regression Discontinuity (RD) Design of Thistlethwaite and Campbell (1960). We organize our discussion of these various research designs by how they secure internal validity: in this view, the RD design can been seen as a close "cousin" of the randomized experiment. An important distinction which emerges from our discussion of "heterogeneous treatment effects" is between ex post (descriptive) and ex ante (predictive) evaluations; these two types of evaluations have distinct, but complementary goals. A second important distinction we make is between statistical statements that are descriptions of our knowledge of the program assignment process and statistical statements that are structural assumptions about individual behavior. Using these distinctions, we examine some commonly employed evaluation strategies, and assess them with a common set of criteria for "internal validity", the foremost goal of an ex post evaluation. In some cases, we also provide some concrete illustrations of how internally valid causal estimates can be supplemented with specific structural assumptions to address "external validity": the estimate from an internally valid "experimental" estimate can be viewed as a "leading term" in an extrapolation for a parameter of interest in an ex ante evaluation.

Suggested Citation

  • John DiNardo & David S. Lee, 2010. "Program Evaluation and Research Designs," NBER Working Papers 16016, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:16016 Note: LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w16016.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Pagan,Adrian & Ullah,Aman, 1999. "Nonparametric Econometrics," Cambridge Books, Cambridge University Press, number 9780521355643, March.
    2. J.J. Heckman & E.E. Leamer (ed.), 2007. "Handbook of Econometrics," Handbook of Econometrics, Elsevier, edition 1, volume 6, number 6b.
    3. Cook, Thomas D., 2008. ""Waiting for Life to Arrive": A history of the regression-discontinuity design in Psychology, Statistics and Economics," Journal of Econometrics, Elsevier, vol. 142(2), pages 636-654, February.
    4. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    5. James J. Heckman & Sergio Urzua & Edward Vytlacil, 2006. "Understanding Instrumental Variables in Models with Essential Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 88(3), pages 389-432, August.
    6. Millimet, Daniel L. & Tchernis, Rusty, 2009. "On the Specification of Propensity Scores, With Applications to the Analysis of Trade Policies," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(3), pages 397-415.
    7. Joshua Angrist & Eric Bettinger & Michael Kremer, 2006. "Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia," American Economic Review, American Economic Association, vol. 96(3), pages 847-862, June.
    8. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    9. Erica Field, 2007. "Entitled to Work: Urban Property Rights and Labor Supply in Peru," The Quarterly Journal of Economics, Oxford University Press, vol. 122(4), pages 1561-1602.
    10. Philip Oreopoulos, 2006. "Estimating Average and Local Average Treatment Effects of Education when Compulsory Schooling Laws Really Matter," American Economic Review, American Economic Association, vol. 96(1), pages 152-175, March.
    11. J.J. Heckman & E.E. Leamer (ed.), 2007. "Handbook of Econometrics," Handbook of Econometrics, Elsevier, edition 1, volume 6, number 6a.
    12. James Heckman & Justin L. Tobias & Edward Vytlacil, 2001. "Four Parameters of Interest in the Evaluation of Social Programs," Southern Economic Journal, Southern Economic Association, vol. 68(2), pages 210-223, October.
    13. Hahn, Jinyong & Todd, Petra & Van der Klaauw, Wilbert, 2001. "Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design," Econometrica, Econometric Society, vol. 69(1), pages 201-209, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Doyle, Joseph J., 2013. "Causal effects of foster care: An instrumental-variables approach," Children and Youth Services Review, Elsevier, vol. 35(7), pages 1143-1151.
    2. Chaisemartin, Clément de, 2014. "Tolerating defiance? Local average treatment effects without monotonicity," CAGE Online Working Paper Series 197, Competitive Advantage in the Global Economy (CAGE).
    3. Bloom, Nicholas & Van Reenen, John, 2011. "Human Resource Management and Productivity," Handbook of Labor Economics, Elsevier.
    4. Baum-Snow, Nathaniel & Ferreira, Fernando, 2015. "Causal Inference in Urban and Regional Economics," Handbook of Regional and Urban Economics, Elsevier.
    5. David E. Card & Pablo Ibarraran & Juan Miguel Villa, 2011. "Building in an Evaluation Component for Active Labor Market Programs: A Practitioner's Guide," SPD Working Papers 1101, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    6. Jens Ludwig & Jeffrey R. Kling & Sendhil Mullainathan, 2011. "Mechanism Experiments and Policy Evaluations," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 17-38, Summer.
    7. Steve Gibbons & Max Nathan & Henry G. Overman, 2014. "Evaluating Spatial Policies," SERC Policy Papers 012, Spatial Economics Research Centre, LSE.
    8. Matthew D. Webb & Arthur Sweetman & Casey Warman, 2016. "Targeting Tax Relief at Youth Employment," Canadian Public Policy, University of Toronto Press, vol. 42(4), pages 415-430, December.
    9. Matthew D. Webb, 2014. "Reworking Wild Bootstrap Based Inference for Clustered Errors," Working Papers 1315, Queen's University, Department of Economics.
    10. repec:taf:jdevst:v:53:y:2017:i:9:p:1358-1375 is not listed on IDEAS
    11. Arni, Patrick, 2012. "Kausale Evaluation von Pilotprojekten: Die Nutzung von Randomisierung in der Praxis," IZA Standpunkte 52, Institute for the Study of Labor (IZA).
    12. Battistin, Erich & De Nadai, Michele & Sianesi, Barbara, 2014. "Misreported schooling, multiple measures and returns to educational qualifications," Journal of Econometrics, Elsevier, vol. 181(2), pages 136-150.
    13. Rasyad A. Parinduri, 2017. "Does Education Improve Health? Evidence from Indonesia," Journal of Development Studies, Taylor & Francis Journals, vol. 53(9), pages 1358-1375, September.
    14. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.
    15. Boockmann Bernhard & Buch Claudia M. & Schnitzer Monika, 2014. "Evidenzbasierte Wirtschaftspolitik in Deutschland: Defizite und Potentiale," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 15(4), pages 307-323, December.
    16. Giulia Faggio, 2014. "Relocation of Public Sector Workers: Evaluating a Place-based Policy," SERC Discussion Papers 0155, Spatial Economics Research Centre, LSE.
    17. Eble,Alex & Boone,Peter & Elbourne,Diana, 2016. "On minimizing the risk of bias in randomized controlled trials in economics," Policy Research Working Paper Series 7746, The World Bank.
    18. Huber, Martin & Wüthrich, Kaspar, 2017. "Evaluating local average and quantile treatment effects under endogeneity based on instruments: a review," FSES Working Papers 479, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    19. de Chaisemartin, Clement, 2013. "Defying the LATE? Identification of local treatment effects when the instrument violates monotonicity," The Warwick Economics Research Paper Series (TWERPS) 1020, University of Warwick, Department of Economics.
    20. Harding, Matthew & Lamarche, Carlos, 2014. "Estimating and testing a quantile regression model with interactive effects," Journal of Econometrics, Elsevier, vol. 178(P1), pages 101-113.

    More about this item

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C50 - Mathematical and Quantitative Methods - - Econometric Modeling - - - General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • H00 - Public Economics - - General - - - General
    • I00 - Health, Education, and Welfare - - General - - - General
    • J00 - Labor and Demographic Economics - - General - - - General
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:16016. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: http://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.