IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2506.11960.html
   My bibliography  Save this paper

Evaluating Program Sequences with Double Machine Learning: An Application to Labor Market Policies

Author

Listed:
  • Fabian Muny

Abstract

Many programs evaluated in observational studies incorporate a sequential structure, where individuals may be assigned to various programs over time. While this complexity is often simplified by analyzing programs at single points in time, this paper reviews, explains, and applies methods for program evaluation within a sequential framework. It outlines the assumptions required for identification under dynamic confounding and demonstrates how extending sequential estimands to dynamic policies enables the construction of more realistic counterfactuals. Furthermore, the paper explores recently developed methods for estimating effects across multiple treatments and time periods, utilizing Double Machine Learning (DML), a flexible estimator that avoids parametric assumptions while preserving desirable statistical properties. Using Swiss administrative data, the methods are demonstrated through an empirical application assessing the participation of unemployed individuals in active labor market policies, where assignment decisions by caseworkers can be reconsidered between two periods. The analysis identifies a temporary wage subsidy as the most effective intervention, on average, even after adjusting for its extended duration compared to other programs. Overall, DML-based analysis of dynamic policies proves to be a useful approach within the program evaluation toolkit.

Suggested Citation

  • Fabian Muny, 2025. "Evaluating Program Sequences with Double Machine Learning: An Application to Labor Market Policies," Papers 2506.11960, arXiv.org.
    • Handle: RePEc:arx:papers:2506.11960
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2506.11960
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    2. Michael Lechner & Stephan Wiehler, 2013. "Does the Order and Timing of Active Labour Market Programmes Matter?," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 75(2), pages 180-212, April.
    3. David Card & Jochen Kluve & Andrea Weber, 2010. "Active Labour Market Policy Evaluations: A Meta-Analysis," Economic Journal, Royal Economic Society, vol. 120(548), pages 452-477, November.
    4. Michael C Knaus & Michael Lechner & Anthony Strittmatter, 2021. "Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence," The Econometrics Journal, Royal Economic Society, vol. 24(1), pages 134-161.
    5. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    6. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    7. Cockx, Bart & Lechner, Michael & Bollens, Joost, 2023. "Priority to unemployed immigrants? A causal machine learning evaluation of training in Belgium," Labour Economics, Elsevier, vol. 80(C).
    8. David Card & Jochen Kluve & Andrea Weber, 2018. "What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations," Journal of the European Economic Association, European Economic Association, vol. 16(3), pages 894-931.
    9. Katharina Dengler, 2019. "Effectiveness of sequences of classroom training for welfare recipients: what works best in West Germany?," Applied Economics, Taylor & Francis Journals, vol. 51(1), pages 1-46, January.
    10. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    11. K. Dengler, 2015. "Effectiveness of sequences of One-Euro-Jobs for welfare recipients in Germany," Applied Economics, Taylor & Francis Journals, vol. 47(57), pages 6170-6190, December.
    12. Stephen Kastoryano & Bas van der Klaauw, 2022. "Dynamic evaluation of job search assistance," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(2), pages 227-241, March.
    13. Gerard J. van den Berg & Johan Vikström, 2022. "Long‐Run Effects of Dynamically Assigned Treatments: A New Methodology and an Evaluation of Training Effects on Earnings," Econometrica, Econometric Society, vol. 90(3), pages 1337-1354, May.
    14. Bruno Crépon & Gerard J. van den Berg, 2016. "Active Labor Market Policies," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 521-546, October.
    15. Michael Lechner & Anthony Strittmatter, 2019. "Practical procedures to deal with common support problems in matching estimation," Econometric Reviews, Taylor & Francis Journals, vol. 38(2), pages 193-207, February.
    16. Bruno Crépon & Marc Ferracci & Grégory Jolivet & Gerard J. van den Berg, 2009. "Active Labor Market Policy Effects in a Dynamic Setting," Journal of the European Economic Association, MIT Press, vol. 7(2-3), pages 595-605, 04-05.
    17. Hugo Bodory & Martin Huber & Lukáš Lafférs, 2022. "Evaluating (weighted) dynamic treatment effects by double machine learning [Identification of causal effects using instrumental variables]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 628-648.
    18. V Chernozhukov & W K Newey & R Singh, 2023. "A simple and general debiased machine learning theorem with finite-sample guarantees," Biometrika, Biometrika Trust, vol. 110(1), pages 257-264.
    19. Xinkun Nie & Emma Brunskill & Stefan Wager, 2021. "Learning When-to-Treat Policies," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(533), pages 392-409, January.
    20. S. A. Murphy, 2003. "Optimal dynamic treatment regimes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 65(2), pages 331-355, May.
    21. van der Laan Mark J. & Rubin Daniel, 2006. "Targeted Maximum Likelihood Learning," The International Journal of Biostatistics, De Gruyter, vol. 2(1), pages 1-40, December.
    22. Baqun Zhang & Anastasios A. Tsiatis & Eric B. Laber & Marie Davidian, 2013. "Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions," Biometrika, Biometrika Trust, vol. 100(3), pages 681-694.
    23. Shakeeb Khan & Elie Tamer, 2010. "Irregular Identification, Support Conditions, and Inverse Weight Estimation," Econometrica, Econometric Society, vol. 78(6), pages 2021-2042, November.
    24. Lechner, Michael, 2009. "Sequential Causal Models for the Evaluation of Labor Market Programs," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 71-83.
    25. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    26. Tran Linh & Yiannoutsos Constantin & Wools-Kaloustian Kara & Siika Abraham & van der Laan Mark & Petersen Maya, 2019. "Double Robust Efficient Estimators of Longitudinal Treatment Effects: Comparative Performance in Simulations and a Case Study," The International Journal of Biostatistics, De Gruyter, vol. 15(2), pages 1-27, November.
    27. Melvin Vooren & Carla Haelermans & Wim Groot & Henriëtte Maassen van den Brink, 2019. "The Effectiveness Of Active Labor Market Policies: A Meta‐Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 33(1), pages 125-149, February.
    28. Barbara Sianesi, 2004. "An Evaluation of the Swedish System of Active Labor Market Programs in the 1990s," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 133-155, February.
    29. Michael Lechner & Ruth Miquel, 2010. "Identification of the effects of dynamic treatments by sequential conditional independence assumptions," Empirical Economics, Springer, vol. 39(1), pages 111-137, August.
    30. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    31. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    32. van der Laan Mark J. & Gruber Susan, 2012. "Targeted Minimum Loss Based Estimation of Causal Effects of Multiple Time Point Interventions," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-41, May.
    33. Victor Chernozhukov & Whitney K. Newey & Rahul Singh, 2022. "Automatic Debiased Machine Learning of Causal and Structural Effects," Econometrica, Econometric Society, vol. 90(3), pages 967-1027, May.
    34. Tran Linh & Yiannoutsos Constantin & Wools-Kaloustian Kara & Siika Abraham & van der Laan Mark & Petersen Maya, 2019. "Double Robust Efficient Estimators of Longitudinal Treatment Effects: Comparative Performance in Simulations and a Case Study," The International Journal of Biostatistics, De Gruyter, vol. 15(2), pages 1-27, November.
    35. Vikström, Johan, 2017. "Dynamic treatment assignment and evaluation of active labor market policies," Labour Economics, Elsevier, vol. 49(C), pages 42-54.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael Lechner, 2023. "Causal Machine Learning and its use for public policy," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 159(1), pages 1-15, December.
    2. Goller, Daniel & Lechner, Michael & Pongratz, Tamara & Wolff, Joachim, 2025. "Active labor market policies for the long-term unemployed: New evidence from causal machine learning," Labour Economics, Elsevier, vol. 94(C).
    3. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    4. Cockx, Bart & Lechner, Michael & Bollens, Joost, 2023. "Priority to unemployed immigrants? A causal machine learning evaluation of training in Belgium," Labour Economics, Elsevier, vol. 80(C).
    5. Hugo Bodory & Martin Huber & Lukáš Lafférs, 2022. "Evaluating (weighted) dynamic treatment effects by double machine learning [Identification of causal effects using instrumental variables]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 628-648.
    6. Jelena Bradic & Weijie Ji & Yuqian Zhang, 2021. "High-dimensional Inference for Dynamic Treatment Effects," Papers 2110.04924, arXiv.org, revised May 2023.
    7. Goller, Daniel & Lechner, Michael & Moczall, Andreas & Wolff, Joachim, 2020. "Does the estimation of the propensity score by machine learning improve matching estimation? The case of Germany's programmes for long term unemployed," Labour Economics, Elsevier, vol. 65(C).
    8. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    9. Arranz, José María & García-Serrano, Carlos, 2024. "You'll never seek alone: The impact of active labour market policies on finding a job," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 58, pages 1-20.
    10. Michael C. Knaus & Michael Lechner & Anthony Strittmatter, 2022. "Heterogeneous Employment Effects of Job Search Programs: A Machine Learning Approach," Journal of Human Resources, University of Wisconsin Press, vol. 57(2), pages 597-636.
    11. Yuqian Zhang & Weijie Ji & Jelena Bradic, 2021. "Dynamic treatment effects: high-dimensional inference under model misspecification," Papers 2111.06818, arXiv.org, revised Jan 2025.
    12. Katharina Dengler, 2019. "Effectiveness of sequences of classroom training for welfare recipients: what works best in West Germany?," Applied Economics, Taylor & Francis Journals, vol. 51(1), pages 1-46, January.
    13. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    14. Tobias Brändle & Lukas Fervers, 2021. "Give it Another Try: What are the Effects of a Job Creation Scheme Especially Designed for Hard-to-Place Workers?," Journal of Labor Research, Springer, vol. 42(3), pages 382-417, December.
    15. Burlat, Héloïse, 2024. "Everybody’s got to learn sometime? A causal machine learning evaluation of training programmes for jobseekers in France," Labour Economics, Elsevier, vol. 89(C).
    16. Feir, Donn. L. & Foley, Kelly & Jones, Maggie E. C., 2022. "Heterogeneous Returns to Active Labour Market Programs for Indigenous Populations," IZA Discussion Papers 15358, Institute of Labor Economics (IZA).
    17. Michael Lechner & Jana Mareckova, 2024. "Comprehensive Causal Machine Learning," Papers 2405.10198, arXiv.org, revised Feb 2025.
    18. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    19. Bernhard Boockmann & Tobias Brändle, 2019. "Coaching, Counseling, Case‐Working: Do They Help the Older Unemployed Out of Benefit Receipt and Back Into the Labor Market?," German Economic Review, Verein für Socialpolitik, vol. 20(4), pages 436-468, November.
    20. Yiyi Huo & Yingying Fan & Fang Han, 2023. "On the adaptation of causal forests to manifold data," Papers 2311.16486, arXiv.org, revised Dec 2023.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2506.11960. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.