IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/6983.html
   My bibliography  Save this paper

The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies

Author

Listed:
  • James J. Heckman
  • Jeffrey A. Smith

Abstract

The key to estimating the impact of a program is constructing the counterfactual outcome representing what would have happened in its absence. This problem becomes more complicated when agents self-select into the program rather than being exogenously assigned to it. This paper uses data from a major social experiment to identify what would have happened to the earnings of self-selected participants in a job training program had they not participated in it. We investigate the implications of these earnings patterns for the validity of widely-used before-after and difference-in-differences estimators. Motivated by the failure of these estimators to produce credible estimates, we investigate the determinants of program participation. We find that labor force status dynamics, rather than earnings or employment dynamics, drive the participation process. Our evidence suggests that training programs often function as a form of job search. Methods that control only for earnings dynamics, like the conventional difference-in-differences estimator, do not adequately capture the underlying differences between participants and non-participants. We use the estimated probabilities of participation in both matching estimators and a nonparametric, conditional version of the differences-in-differences estimator and produce large reductions in the selection bias in non-experimental estimates of the effect of training on earnings.

Suggested Citation

  • James J. Heckman & Jeffrey A. Smith, 1999. "The Pre-Program Earnings Dip and the Determinants of Participation in a Social Program: Implications for Simple Program Evaluation Strategies," NBER Working Papers 6983, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:6983
    Note: LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w6983.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    2. Katherine P. Dickinson & Terry R. Johnson & Richard W. West, 1987. "An Analysis of the Sensitivity of Quasi-Experimental Net Impact Estimates of Ceta Programs," Evaluation Review, , vol. 11(4), pages 452-472, August.
    3. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 605-654.
    4. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    5. Edward C. Bryant & Kalman Rupp, 1987. "Evaluating the Impact of Ceta On Participant Earnings," Evaluation Review, , vol. 11(4), pages 473-492, August.
    6. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    7. Heckman, James J, 1978. "Dummy Endogenous Variables in a Simultaneous Equation System," Econometrica, Econometric Society, vol. 46(4), pages 931-959, July.
    8. Bassi, Laurie J, 1984. "Estimating the Effect of Training Programs with Non-Random Selection," The Review of Economics and Statistics, MIT Press, vol. 66(1), pages 36-43, February.
    9. James J. Heckman & Jeffrey A. Smith, 2004. "The Determinants of Participation in a Social Program: Evidence from a Prototypical Job Training Program," Journal of Labor Economics, University of Chicago Press, vol. 22(2), pages 243-298, April.
    10. Ashenfelter, Orley & Card, David, 1985. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs," The Review of Economics and Statistics, MIT Press, vol. 67(4), pages 648-660, November.
    11. James J. Heckman & Jeffrey A. Smith, 1998. "Evaluating the Welfare State," NBER Working Papers 6542, National Bureau of Economic Research, Inc.
    12. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    13. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    14. Hector Chade & Gustavo Ventura, 1998. "Taxes and Marriage: A Two-Sided Search Analysis," UWO Department of Economics Working Papers 9819, University of Western Ontario, Department of Economics.
    15. Heckman, James J. & Robb, Richard Jr., 1985. "Alternative methods for evaluating the impact of interventions : An overview," Journal of Econometrics, Elsevier, vol. 30(1-2), pages 239-267.
    16. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    17. James Heckman, 1997. "Instrumental Variables: A Study of Implicit Behavioral Assumptions Used in Making Program Evaluations," Journal of Human Resources, University of Wisconsin Press, vol. 32(3), pages 441-462.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    2. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    3. James J. Heckman, 2005. "Micro Data, Heterogeneity and the Evaluation of Public Policy Part 2," The American Economist, Sage Publications, vol. 49(1), pages 16-44, March.
    4. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    5. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    6. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    7. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
    8. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    9. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    10. Hujer, Reinhard & Wellner, Marc, 2000. "The Effects of Public Sector Sponsored Training on Individual Employment Performance in East Germany," IZA Discussion Papers 141, Institute of Labor Economics (IZA).
    11. Almus, Matthias, 2001. "Evaluating the impact of public start-up assistance: results from an econometric approach," ZEW Discussion Papers 01-23, ZEW - Leibniz Centre for European Economic Research.
    12. Stephan Thomsen, 2009. "Job Search Assistance Programs in Europe: Evaluation Methods and Recent Empirical Findings," FEMM Working Papers 09018, Otto-von-Guericke University Magdeburg, Faculty of Economics and Management.
    13. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    14. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    15. Regner, Hakan, 2002. "A nonexperimental evaluation of training programs for the unemployed in Sweden," Labour Economics, Elsevier, vol. 9(2), pages 187-206, April.
    16. Guo, Shenyang & Barth, Richard P. & Gibbons, Claire, 2006. "Propensity score matching strategies for evaluating substance abuse services for child welfare clients," Children and Youth Services Review, Elsevier, vol. 28(4), pages 357-383, April.
    17. James J. Heckman & Jeffrey A. Smith, 2004. "The Determinants of Participation in a Social Program: Evidence from a Prototypical Job Training Program," Journal of Labor Economics, University of Chicago Press, vol. 22(2), pages 243-298, April.
    18. Gustavo Canavire-Bacarreza & Luis Castro Peñarrieta & Darwin Ugarte Ontiveros, 2021. "Outliers in Semi-Parametric Estimation of Treatment Effects," Econometrics, MDPI, Open Access Journal, vol. 9(2), pages 1-32, April.
    19. Kluve, Jochen & Lehmann, Hartmut & Schmidt, Christoph M., 2008. "Disentangling Treatment Effects of Active Labor Market Policies: The Role of Labor Force Status Sequences," Labour Economics, Elsevier, vol. 15(6), pages 1270-1295, December.
    20. Michael Lechner, 2000. "An Evaluation of Public-Sector-Sponsored Continuous Vocational Training Programs in East Germany," Journal of Human Resources, University of Wisconsin Press, vol. 35(2), pages 347-375.

    More about this item

    JEL classification:

    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:6983. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.