IDEAS home Printed from https://ideas.repec.org/a/taf/japsta/v30y2003i8p893-907.html
   My bibliography  Save this article

Impact evaluation of job training programmes: Selection bias in multilevel models

Author

Listed:
  • R. Bellio
  • E. Gori

Abstract

This paper focuses on the evaluation of a job training programme composed of several different courses. The aim is to evaluate the impact of the programme for the participants with respect to non-participants, paying attention to possible differences in the effectiveness between the courses. The analysis is based on discrete data with a hierarchical structure. Multilevel modelling is the natural choice in this setting, but the results may be severely affected by selection bias. We propose a two-step procedure, which suits both the hierarchical structure and the observational nature of data. The method selects the appropriate control group, using standard results of the propensity score methodology. A suitable multilevel model is formulated, and the dependence of the results on the amount of non-random sample selection is analysed within a likelihood-based framework. As a result, rankings for comparative performances are obtained, adjusted for the amount of plausible selection bias. The procedure is illustrated with reference to a data set about a job training programme organized in Italy in the late 1990s.

Suggested Citation

  • R. Bellio & E. Gori, 2003. "Impact evaluation of job training programmes: Selection bias in multilevel models," Journal of Applied Statistics, Taylor & Francis Journals, vol. 30(8), pages 893-907.
    • Handle: RePEc:taf:japsta:v:30:y:2003:i:8:p:893-907
    DOI: 10.1080/0266476032000075976
    as

    Download full text from publisher

    File URL: http://www.tandfonline.com/doi/abs/10.1080/0266476032000075976
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/0266476032000075976?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    2. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    3. Harvey Goldstein & Michael J. R. Healy, 1995. "The Graphical Presentation of a Collection of Means," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(1), pages 175-177, January.
    4. Vella, Francis & Verbeek, Marno, 1999. "Estimating and Interpreting Models with Endogenous Treatment Effects," Journal of Business & Economic Statistics, American Statistical Association, vol. 17(4), pages 473-478, October.
    5. Angrist, Joshua D, 2001. "Estimations of Limited Dependent Variable Models with Dummy Endogenous Regressors: Simple Strategies for Empirical Practice," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(1), pages 2-16, January.
    6. Carrasco, Raquel, 2001. "Binary Choice with Binary Endogenous Regressors in Panel Data: Estimating the Effect of Fertility on Female Labor Participation," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 385-394, October.
    7. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    8. Lee, Lung-Fei, 1983. "Generalized Econometric Models with Selectivity," Econometrica, Econometric Society, vol. 51(2), pages 507-512, March.
    9. John Copas & Shinto Eguchi, 2001. "Local sensitivity approximations for selectivity bias," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(4), pages 871-895.
    10. Ian Plewis, 2002. "Modelling impact heterogeneity," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 31-38, February.
    11. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    12. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82, February.
    13. Harvey Goldstein & Sally Thomas, 1996. "Using Examination Results as Indicators of School and College Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(1), pages 149-163, January.
    14. Guido W. Imbens, 1999. "The Role of the Propensity Score in Estimating Dose-Response Functions," NBER Technical Working Papers 0237, National Bureau of Economic Research, Inc.
    15. D. Firth & C. Payne & J. Payne, 1999. "Efficacy of programmes for the unemployed: discrete time modelling of duration data from a matched‐comparison study," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(1), pages 111-120.
    16. Angrist, Joshua D, 2001. "Estimations of Limited Dependent Variable Models with Dummy Endogenous Regressors: Simple Strategies for Empirical Practice: Reply," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(1), pages 27-28, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hajime Seya & Junyi Zhang & Makoto Chikaraishi & Ying Jiang, 2020. "Decisions on truck parking place and time on expressways: an analysis using digital tachograph data," Transportation, Springer, vol. 47(2), pages 555-583, April.
    2. Greta Falavigna & Elena Ragazzi & Lisa Sella, 2014. "Gender inequalities and labour integration. An integrated approach to vocational training in Piedmont," CERIS Working Paper 201407, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    3. Elena Ragazzi & Lisa Sella, 2013. "Una valutazione di impatto delle politiche formative regionali: il caso piemontese," CERIS Working Paper 201315, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    4. Elena Ragazzi & Lisa Sella, 2013. "Migration and work: the cohesive role of vocational training policies," CERIS Working Paper 201316, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    5. Leonardo Grilli & Carla Rampichini, 2007. "A multilevel multinomial logit model for the analysis of graduates’ skills," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 16(3), pages 381-393, November.
    6. Elena Ragazzi & Lisa Sella, 2013. "The effectiveness of vocational training policies: methods for an impact evaluation," CERIS Working Paper 201314, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    7. Leonardo Grilli & Carla Rampichini, 2010. "Selection bias in linear mixed models," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(3), pages 309-329.
    8. Falavigna Falavigna & Elena Ragazzi & Lisa Sella, "undated". "Vocational training and labour market: inclusion or segregation paths? An integrated approach on immigrant trainees in Piedmont," CERIS Working Paper 201425, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael Lechner, 2002. "Mikroökonometrische Evaluation arbeitsmarktpolitischer Massnahmen," University of St. Gallen Department of Economics working paper series 2002 2002-20, Department of Economics, University of St. Gallen.
    2. Hämäläinen, Kari & Ollikainen, Virve, 2004. "Differential Effects of Active Labour Market Programmes in the Early Stages of Young People's Unemployment," Research Reports 115, VATT Institute for Economic Research.
    3. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    4. Flores, Carlos A. & Mitnik, Oscar A., 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," IZA Discussion Papers 4451, Institute of Labor Economics (IZA).
    5. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    6. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    7. Fernández-Val, Iván & Vella, Francis, 2011. "Bias corrections for two-step fixed effects panel data estimators," Journal of Econometrics, Elsevier, vol. 163(2), pages 144-162, August.
    8. Emilia Soldani, 2021. "Public kindergarten, maternal labor supply, and earnings in the longer run: Too little too late?," LABOUR, CEIS, vol. 35(2), pages 214-263, June.
    9. Bryson, Alex & Dorsett, Richard & Purdon, Susan, 2002. "The use of propensity score matching in the evaluation of active labour market policies," LSE Research Online Documents on Economics 4993, London School of Economics and Political Science, LSE Library.
    10. Denis Conniffe & Vanessa Gash & Philip J., 2000. "Evaluating Programmes: Experiments, Non-Experiments and Propensity Scores," Papers WP126, Economic and Social Research Institute (ESRI).
    11. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    12. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    13. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
    14. Ichimura, Hidehiko & Todd, Petra E., 2007. "Implementing Nonparametric and Semiparametric Estimators," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 74, Elsevier.
    15. René Algesheimer & Sharad Borle & Utpal M. Dholakia & Siddharth S. Singh, 2010. "The Impact of Customer Community Participation on Customer Behaviors: An Empirical Investigation," Marketing Science, INFORMS, vol. 29(4), pages 756-769, 07-08.
    16. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    17. Denis Conniffe & Vanessa Gash & Philip J. O'Connell, 2000. "Evaluating State Programmes - “Natural Experiments” and Propensity Scores," The Economic and Social Review, Economic and Social Studies, vol. 31(4), pages 283-308.
    18. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    19. Jie Wei & Yonghui Zhang, 2022. "Panel Probit Models with Time‐Varying Individual Effects: Reestimating the Effects of Fertility on Female Labour Participation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 84(4), pages 799-829, August.
    20. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:japsta:v:30:y:2003:i:8:p:893-907. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/CJAS20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.