IDEAS home Printed from https://ideas.repec.org/p/iab/iabdpa/202005.html
   My bibliography  Save this paper

Does the estimation of the propensity score by machine learning improve matching estimation? : The case of Germany's programmes for long term unemployed

Author

Listed:
  • Goller, Daniel

    (University of St.Gallen)

  • Lechner, Michael

    (University of St.Gallen)

  • Moczall, Andreas

    (Institute for Employment Research (IAB), Nuremberg, Germany)

  • Wolff, Joachim

    (Institute for Employment Research (IAB), Nuremberg, Germany)

Abstract

"Matching-type estimators using the propensity score are the major workhorse in active labour market policy evaluation. This work investigates if machine learning algorithms for estimating the propensity score lead to more credible estimation of average treatment effects on the treated using a radius matching framework. Considering two popular methods, the results are ambiguous: We find that using LASSO based logit models to estimate the propensity score delivers more credible results than conventional methods in small and medium sized high dimensional datasets. However, the usage of Random Forests to estimate the propensity score may lead to a deterioration of the performance in situations with a low treatment share. The application reveals a positive effect of the training programme on days in employment for long-term unemployed. While the choice of the 'first stage' is highly relevant for settings with low number of observations and few treated, machine learning and conventional estimation becomes more similar in larger samples and higher treatment shares." (Author's abstract, IAB-Doku) ((en))

Suggested Citation

  • Goller, Daniel & Lechner, Michael & Moczall, Andreas & Wolff, Joachim, 2020. "Does the estimation of the propensity score by machine learning improve matching estimation? : The case of Germany's programmes for long term unemployed," IAB-Discussion Paper 202005, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    • Handle: RePEc:iab:iabdpa:202005
    as

    Download full text from publisher

    File URL: https://doku.iab.de/discussionpapers/2020/dp0520.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    2. Goller, Daniel & Krumer, Alex, 2019. "Let’s meet as usual: Do games on non-frequent days differ? Evidence from top European soccer leagues," Economics Working Paper Series 1907, University of St. Gallen, School of Economics and Political Science.
    3. Martin Biewen & Bernd Fitzenberger & Aderonke Osikominu & Marie Paul, 2014. "The Effectiveness of Public-Sponsored Training Revisited: The Importance of Data and Methodological Choices," Journal of Labor Economics, University of Chicago Press, vol. 32(4), pages 837-897.
    4. Lechner, Michael, 2018. "Modified Causal Forests for Estimating Heterogeneous Causal Effects," IZA Discussion Papers 12040, Institute of Labor Economics (IZA).
    5. Alberto Abadie & Guido W. Imbens, 2016. "Matching on the Estimated Propensity Score," Econometrica, Econometric Society, vol. 84, pages 781-807, March.
    6. Michael Lechner & Conny Wunsch, 2009. "Are Training Programs More Effective When Unemployment Is High?," Journal of Labor Economics, University of Chicago Press, vol. 27(4), pages 653-692, October.
    7. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    8. Michael Lechner & Anthony Strittmatter, 2019. "Practical procedures to deal with common support problems in matching estimation," Econometric Reviews, Taylor & Francis Journals, vol. 38(2), pages 193-207, February.
    9. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "Inference on Treatment Effects after Selection among High-Dimensional Controlsâ€," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(2), pages 608-650.
    10. David Card & Jochen Kluve & Andrea Weber, 2018. "What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations," Journal of the European Economic Association, European Economic Association, vol. 16(3), pages 894-931.
    11. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    12. Michael C Knaus & Michael Lechner & Anthony Strittmatter, 2021. "Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence," The Econometrics Journal, Royal Economic Society, vol. 24(1), pages 134-161.
    13. Tamara Harrer & Andreas Moczall & Joachim Wolff, 2020. "Free, free, set them free? Are programmes effective that allow job centres considerable freedom to choose the exact design?," International Journal of Social Welfare, John Wiley & Sons, vol. 29(2), pages 154-167, April.
    14. Annabelle Doerr & Bernd Fitzenberger & Thomas Kruppe & Marie Paul & Anthony Strittmatter, 2017. "Employment and Earnings Effects of Awarding Training Vouchers in Germany," ILR Review, Cornell University, ILR School, vol. 70(3), pages 767-812, May.
    15. Michael Lechner & Ruth Miquel & Conny Wunsch, 2011. "Long‐Run Effects Of Public Sector Sponsored Training In West Germany," Journal of the European Economic Association, European Economic Association, vol. 9(4), pages 742-784, August.
    16. Conny Wunsch & Michael Lechner, 2008. "What Did All the Money Do? On the General Ineffectiveness of Recent West German Labour Market Programmes," Kyklos, Wiley Blackwell, vol. 61(1), pages 134-174, February.
    17. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    18. Guido W. Imbens, 2015. "Matching Methods in Practice: Three Examples," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 373-419.
    19. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    20. Sebastian Calónico & Jeffrey Smith, 2017. "The Women of the National Supported Work Demonstration," Journal of Labor Economics, University of Chicago Press, vol. 35(S1), pages 65-97.
    21. Athey, Susan & Imbens, Guido W., 2019. "Machine Learning Methods Economists Should Know About," Research Papers 3776, Stanford University, Graduate School of Business.
    22. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    23. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    24. Martin Huber & Michael Lechner & Andreas Steinmayr, 2015. "Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour," Empirical Economics, Springer, vol. 49(1), pages 1-31, August.
    25. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    26. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    27. Susan Athey & Guido W. Imbens, 2019. "Machine Learning Methods That Economists Should Know About," Annual Review of Economics, Annual Reviews, vol. 11(1), pages 685-725, August.
    28. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    29. Alex Krumer & Michael Lechner, 2018. "Midweek Effect On Soccer Performance: Evidence From The German Bundesliga," Economic Inquiry, Western Economic Association International, vol. 56(1), pages 193-207, January.
    30. Joseph Antonelli & Matthew Cefalu & Nathan Palmer & Denis Agniel, 2018. "Doubly robust matching estimators for high dimensional confounding adjustment," Biometrics, The International Biometric Society, vol. 74(4), pages 1171-1179, December.
    31. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    32. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2014. "Unobservable, but Unimportant? The Influence of Personality Traits (and Other Usually Unobserved Variables) for the Evaluation of Labor Market Policies," IZA Discussion Papers 8337, Institute of Labor Economics (IZA).
    33. Goller, Daniel & Krumer, Alex, 2020. "Let's meet as usual: Do games played on non-frequent days differ? Evidence from top European soccer leagues," European Journal of Operational Research, Elsevier, vol. 286(2), pages 740-754.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Caron, Laura & Tiongson, Erwin R., 2022. "Households in Transit: COVID-19 and the Changing Measurement of Welfare," IZA Discussion Papers 15670, Institute of Labor Economics (IZA).
    2. Hoai An Le Thi & Manh Cuong Nguyen, 2017. "DCA based algorithms for feature selection in multi-class support vector machine," Annals of Operations Research, Springer, vol. 249(1), pages 273-300, February.
    3. Black, Dan A. & Grogger, Jeffrey & Kirchmaier, Tom & Sanders, Koen, 2023. "Criminal charges, risk assessment and violent recidivism in cases of domestic abuse," LSE Research Online Documents on Economics 121374, London School of Economics and Political Science, LSE Library.
    4. Goller, Daniel & Harrer, Tamara & Lechner, Michael & Wolff, Joachim, 2021. "Active labour market policies for the long-term unemployed: New evidence from causal machine learning," Economics Working Paper Series 2108, University of St. Gallen, School of Economics and Political Science.
    5. Heigle, Julia & Pfeiffer, Friedhelm, 2020. "Langfristige Wirkungen eines nicht abgeschlossenen Studiums auf individuelle Arbeitsmarktergebnisse und die allgemeine Lebenszufriedenheit," ZEW Discussion Papers 20-004, ZEW - Leibniz Centre for European Economic Research.
    6. Felipe Barrera-Osorio & Paul Gertler & Nozomi Nakajima & Harry Patrinos, 2020. "Promoting Parental Involvement in Schools: Evidence From Two Randomized Experiments," NBER Working Papers 28040, National Bureau of Economic Research, Inc.
    7. Cuiqing Jiang & Zhao Wang & Ruiya Wang & Yong Ding, 2018. "Loan default prediction by combining soft information extracted from descriptive text in online peer-to-peer lending," Annals of Operations Research, Springer, vol. 266(1), pages 511-529, July.
    8. Cappelletti, Matilde & Giuffrida, Leonardo M., 2022. "Targeted bidders in government tenders," ZEW Discussion Papers 22-030, ZEW - Leibniz Centre for European Economic Research.
    9. Ahrens, Achim & Hansen, Christian B. & Schaffer, Mark E & Wiemann, Thomas, 2024. "Model Averaging and Double Machine Learning," IZA Discussion Papers 16714, Institute of Labor Economics (IZA).
    10. Joshua D. Angrist & Brigham Frandsen, 2022. "Machine Labor," Journal of Labor Economics, University of Chicago Press, vol. 40(S1), pages 97-140.
    11. Alena Bömmel & Song Song & Piotr Majer & Peter Mohr & Hauke Heekeren & Wolfgang Härdle, 2014. "Risk Patterns and Correlated Brain Activities. Multidimensional Statistical Analysis of fMRI Data in Economic Decision Making Study," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 489-514, July.
    12. Goller, Daniel & Krumer, Alex, 2020. "Let's meet as usual: Do games played on non-frequent days differ? Evidence from top European soccer leagues," European Journal of Operational Research, Elsevier, vol. 286(2), pages 740-754.
    13. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    14. Börschlein, Benjamin & Bossler, Mario, 2021. "A new machine learning-based treatment bite for long run minimum wage evaluations," VfS Annual Conference 2021 (Virtual Conference): Climate Economics 242441, Verein für Socialpolitik / German Economic Association.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Seonho Shin, 2022. "Evaluating the Effect of the Matching Grant Program for Refugees: An Observational Study Using Matching, Weighting, and the Mantel-Haenszel Test," Journal of Labor Research, Springer, vol. 43(1), pages 103-133, March.
    2. Daniel Goller & Tamara Harrer & Michael Lechner & Joachim Wolff, 2021. "Active labour market policies for the long-term unemployed: New evidence from causal machine learning," Papers 2106.10141, arXiv.org, revised May 2023.
    3. Michael Lechner, 2023. "Causal Machine Learning and its use for public policy," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 159(1), pages 1-15, December.
    4. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    5. Cockx, Bart & Lechner, Michael & Bollens, Joost, 2023. "Priority to unemployed immigrants? A causal machine learning evaluation of training in Belgium," Labour Economics, Elsevier, vol. 80(C).
    6. Michael Lechner & Anthony Strittmatter, 2019. "Practical procedures to deal with common support problems in matching estimation," Econometric Reviews, Taylor & Francis Journals, vol. 38(2), pages 193-207, February.
    7. Susan Athey & Guido W. Imbens & Jonas Metzger & Evan M. Munro, 2019. "Using Wasserstein Generative Adversarial Networks for the Design of Monte Carlo Simulations," NBER Working Papers 26566, National Bureau of Economic Research, Inc.
    8. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    9. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    10. Doerr, Annabelle, 2017. "Back to work: The Long-term Effects of Vocational Training for Female Job Returners," VfS Annual Conference 2017 (Vienna): Alternative Structures for Money and Banking 168213, Verein für Socialpolitik / German Economic Association.
    11. Martin Huber & Michael Lechner & Andreas Steinmayr, 2015. "Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour," Empirical Economics, Springer, vol. 49(1), pages 1-31, August.
    12. Hugo Bodory & Lorenzo Camponovo & Martin Huber & Michael Lechner, 2020. "The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 183-200, January.
    13. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    14. Hagen, Tobias, 2016. "Econometric evaluation of a placement coaching program for recipients of disability insurance benefits in Switzerland," Working Paper Series 10, Frankfurt University of Applied Sciences, Faculty of Business and Law.
    15. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    16. Tobias Brändle & Lukas Fervers, 2021. "Give it Another Try: What are the Effects of a Job Creation Scheme Especially Designed for Hard-to-Place Workers?," Journal of Labor Research, Springer, vol. 42(3), pages 382-417, December.
    17. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    18. Zhexiao Lin & Fang Han, 2022. "On regression-adjusted imputation estimators of the average treatment effect," Papers 2212.05424, arXiv.org, revised Jan 2023.
    19. Michael C. Knaus, 2021. "A double machine learning approach to estimate the effects of musical practice on student’s skills," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(1), pages 282-300, January.
    20. Daniel Goller, 2023. "Analysing a built-in advantage in asymmetric darts contests using causal machine learning," Annals of Operations Research, Springer, vol. 325(1), pages 649-679, June.

    More about this item

    Keywords

    Bundesrepublik Deutschland ; Beschäftigungseffekte ; Propensity Score Matching ; Algorithmus ; künstliche Intelligenz ; Langzeitarbeitslose ; Schätzung ; Trainingsmaßnahme ; arbeitsmarktpolitische Maßnahme ; Wirkungsforschung ; 2009-2010;
    All these keywords.

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • J68 - Labor and Demographic Economics - - Mobility, Unemployment, Vacancies, and Immigrant Workers - - - Public Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iab:iabdpa:202005. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: IAB, Geschäftsbereich Wissenschaftliche Fachinformation und Bibliothek (email available below). General contact details of provider: https://edirc.repec.org/data/iabbbde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.