IDEAS home Printed from https://ideas.repec.org/p/mia/wpaper/2011-10.html
   My bibliography  Save this paper

Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies

Author

Listed:
  • Carlos A. Flores

    (Department of Economics, University of Miami)

  • Oscar A. Mitnik

    (Department of Economics, University of Miami)

Abstract

We consider the problem of using data from several programs, each implemented at a different location, to compare what their effect would be if they were implemented at a specific location. In particular, we study the effectiveness of nonexperimental strategies in adjusting for differences across comparison groups arising from two sources. First, we adjust for differences in the distribution of individual characteristics simultaneously across all locations by using unconfoundedness-based and conditional difference-in-difference methods for multiple treatments. Second, we explicitly adjust for differences in local economic conditions. We stress the importance of analyzing the overlap of, and adjusting for, local economic conditions after program participation. Our results suggest that the strategies studied are valuable econometric tools for the problem we consider, as long as we adjust for a rich set of individual characteristics and have sufficient overlap across locations for both individual and local labor market characteristics. Our results show that the overlap analysis of these two sets of variables is critical for identifying non-comparable groups and they illustrate the difficulty of adjusting for local economic conditions that differ greatly across locations.

Suggested Citation

  • Carlos A. Flores & Oscar A. Mitnik, 2011. "Comparing Treatments across Labor Markets: An Assessment of Nonexperimental Multiple-Treatment Strategies," Working Papers 2011-10, University of Miami, Department of Economics.
    • Handle: RePEc:mia:wpaper:2011-10
    as

    Download full text from publisher

    File URL: https://www.herbert.miami.edu/_assets/files/repec/wp2011-10.pdf
    File Function: First version, 2011
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82, February.
    2. James J. Heckman & Jeffrey Smith, 2000. "The Sensitivity of Experimental Impact Estimates (Evidence from the National JTPA Study)," NBER Chapters, in: Youth Employment and Joblessness in Advanced Countries, pages 331-356, National Bureau of Economic Research, Inc.
    3. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
    4. Markus Frölich & Almas Heshmati & Michael Lechner, 2004. "A microeconometric evaluation of rehabilitation of long-term sickness in Sweden," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(3), pages 375-396.
    5. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    6. Markus Frölich, 2004. "Programme Evaluation with Multiple Treatments," Journal of Economic Surveys, Wiley Blackwell, vol. 18(2), pages 181-224, April.
    7. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
    8. Alfonso Flores‐Lagunes & Arturo Gonzalez & Todd Neumann, 2010. "Learning But Not Earning? The Impact Of Job Corps Training On Hispanic Youth," Economic Inquiry, Western Economic Association International, vol. 48(3), pages 651-667, July.
    9. Andrew Dyke & Carolyn J. Heinrich & Peter R. Mueser & Kenneth R. Troske & Kyung-Seong Jeon, 2006. "The Effects of Welfare-to-Work Program Activities on Labor Market Outcomes," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 567-608, July.
    10. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    11. David Greenberg & Philip K. Robins, 2011. "Have Welfare-to-Work Programs Improved over Time in Putting Welfare Recipients to Work?," ILR Review, Cornell University, ILR School, vol. 64(5), pages 910-920, October.
    12. David Greenberg & Philip K. Robins, 2011. "Have Welfare-to-Work Programs Improved over Time in Putting Welfare Recipients to Work?," ILR Review, Cornell University, ILR School, vol. 64(5), pages 910-920, October.
    13. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
    14. Miana Plesca & Jeffrey Smith, 2008. "Evaluating multi-treatment programs: theory and evidence from the U.S. Job Training Partnership Act experiment," Studies in Empirical Economics, in: Christian Dustmann & Bernd Fitzenberger & Stephen Machin (ed.), The Economics of Education and Training, pages 293-330, Springer.
    15. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    16. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-937, September.
    17. Guido W. Imbens, 1999. "The Role of the Propensity Score in Estimating Dose-Response Functions," NBER Technical Working Papers 0237, National Bureau of Economic Research, Inc.
    18. James Heckman & Neil Hohmann & Jeffrey Smith & Michael Khoo, 2000. "Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 115(2), pages 651-694.
    19. McCrary, Justin, 2008. "Manipulation of the running variable in the regression discontinuity design: A density test," Journal of Econometrics, Elsevier, vol. 142(2), pages 698-714, February.
    20. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    21. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute of Labor Economics (IZA).
    22. Shakeeb Khan & Elie Tamer, 2010. "Irregular Identification, Support Conditions, and Inverse Weight Estimation," Econometrica, Econometric Society, vol. 78(6), pages 2021-2042, November.
    23. V. Joseph Hotz & Guido W. Imbens & Julie H. Mortimer, 1999. "Predicting the Efficacy of Future Training Programs Using Past Experiences," NBER Technical Working Papers 0238, National Bureau of Economic Research, Inc.
    24. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    25. Bryan S. Graham & Cristine Campos De Xavier Pinto & Daniel Egel, 2012. "Inverse Probability Tilting for Moment Condition Models with Missing Data," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 1053-1079.
    26. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    27. Newey, Whitney K., 1994. "Kernel Estimation of Partial Means and a General Variance Estimator," Econometric Theory, Cambridge University Press, vol. 10(2), pages 1-21, June.
    28. David G. Blanchflower & Richard B. Freeman, 2000. "Youth Employment and Joblessness in Advanced Countries," NBER Books, National Bureau of Economic Research, Inc, number blan00-1, March.
    29. Dehejia, Rajeev H, 2003. "Was There a Riverside Miracle? A Hierarchical Framework for Evaluating Programs with Grouped Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 1-11, January.
    30. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    31. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 156-179, February.
    32. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    33. Dolton, Peter & Smith, Jeffrey A., 2011. "The Impact of the UK New Deal for Lone Parents on Benefit Receipt," IZA Discussion Papers 5491, Institute of Labor Economics (IZA).
    34. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics,, Palgrave Macmillan.
    35. Wang-Sheng Lee, 2013. "Propensity score matching and variations on the balancing test," Empirical Economics, Springer, vol. 44(1), pages 47-80, February.
    36. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marisa Coetzee, 2013. "Finding the Benefits: Estimating the Impact of The South African Child Support Grant," South African Journal of Economics, Economic Society of South Africa, vol. 81(3), pages 427-450, September.
    2. Ay, Jean-Sauveur & Le Gallo, Julie, 2021. "The Signaling Values of Nested Wine Names," Working Papers 321851, American Association of Wine Economists.
    3. Uchino, Taisuke & Uesugi, Iichiro, 2022. "The effects of a megabank merger on firm-Bank relationships and loan availability☆," Journal of the Japanese and International Economies, Elsevier, vol. 63(C).
    4. Takeshima, Hiroyuki & Adhikari, Rajendra Prasad & Shivakoti, Sabnam & Kaphle, Basu Dev & Kumar, Anjani, 2017. "Heterogeneous returns to chemical fertilizer at the intensive margins: Insights from Nepal," Food Policy, Elsevier, vol. 69(C), pages 97-109.
    5. Aparicio, Gabriela & Bobic, Vida & De Olloqui, Fernando & Carmen, María & Diez, María Carmen Fernández & Gerardino, Maria Paula & Mitnik, Oscar A. & Macedo, Sebastian Vargas, 2021. "Liquidity or Capital? The Impacts of Easing Credit Constraints in Rural Mexico," IZA Discussion Papers 14477, Institute of Labor Economics (IZA).
    6. Laborda, Leopoldo & Mejalenko, Juan & Gómez-Veiga, Isabel, 2023. "Bilingualism and intelligence in children exposed to poverty environments: A Raven's error pattern analysis using a generalized propensity score method," Intelligence, Elsevier, vol. 98(C).
    7. Bogaard, Hein & Svejnar, Jan, 2018. "Incentive pay and performance: Insider econometrics in a multi-unit firm," Labour Economics, Elsevier, vol. 54(C), pages 100-115.
    8. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    9. Chung Choe & Alfonso Flores-Lagunes & Sang-Jun Lee, 2015. "Do dropouts with longer training exposure benefit from training programs? Korean evidence employing methods for continuous treatments," Empirical Economics, Springer, vol. 48(2), pages 849-881, March.
    10. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    11. Finn McGuire & Noemi Kreif & Peter C. Smith, 2021. "The effect of distance on maternal institutional delivery choice: Evidence from Malawi," Health Economics, John Wiley & Sons, Ltd., vol. 30(9), pages 2144-2167, September.
    12. Hiroyuki Takeshima & Kamiljon Akramov & Allen Park & Jarilkasin Ilyasov & Tanzila Ergasheva, 2022. "Agriculture-Nutrition Linkages, Cooking-Time, Intrahousehold Equality Among Women and Children: Evidence from Tajikistan," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 34(2), pages 940-977, April.
    13. Takeshima, Hiroyuki & Nasir, Abdullahi Mohammed, 2017. "The role of the locations of public sector varietal development activities on agricultural productivity: Evidence from northern Nigeria:," NSSP working papers 42, International Food Policy Research Institute (IFPRI).
    14. Adesina, Adedoyin & Akogun, Oladele & Dillon, Andrew & Friedman, Jed & Njobdi, Sani & Serneels, Pieter, 2017. "Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?," 2018 Allied Social Sciences Association (ASSA) Annual Meeting, January 5-7, 2018, Philadelphia, Pennsylvania 266292, Agricultural and Applied Economics Association.
    15. Hiroyuki Takeshima, 2019. "Geography of plant breeding systems, agroclimatic similarity, and agricultural productivity: evidence from Nigeria," Agricultural Economics, International Association of Agricultural Economists, vol. 50(1), pages 67-78, January.
    16. Das, Ashis & Friedman, Jed & Kandpal, Eeshani, 2014. "Does involvement of local NGOs enhance public service delivery ? cautionary evidence from a Malaria-prevention evaluation in India," Policy Research Working Paper Series 6931, The World Bank.
    17. Dehejia Rajeev, 2015. "Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 47-69, June.
    18. Salgado, Edgar & Mitnik, Oscar A., 2021. "Spatial and Time Spillovers of Driving Restrictions: Causal Evidence from Limas Pico y Placa Policy," IDB Publications (Working Papers) 11818, Inter-American Development Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Flores, Carlos A. & Mitnik, Oscar A., 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," IZA Discussion Papers 4451, Institute of Labor Economics (IZA).
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Dolton, Peter & Smith, Jeffrey A., 2011. "The Impact of the UK New Deal for Lone Parents on Benefit Receipt," IZA Discussion Papers 5491, Institute of Labor Economics (IZA).
    4. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the California GAIN Program," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 521-566, July.
    5. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    6. Flores-Lagunes, Alfonso & Gonzalez, Arturo & Neumann, Todd C., 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," IZA Discussion Papers 2846, Institute of Labor Economics (IZA).
    7. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    8. Carlson, Deven & Haveman, Robert & Kaplan, Tom & Wolfe, Barbara, 2012. "Long-term earnings and employment effects of housing voucher receipt," Journal of Urban Economics, Elsevier, vol. 71(1), pages 128-150.
    9. Robert Haveman & Barbara Wolfe, 2012. "Long-Term Effects of Public Low-Income Housing Vouchers: Work, Neighborhood, Family Composition and Childcare Usage," CEPR Discussion Papers 667, Centre for Economic Policy Research, Research School of Economics, Australian National University.
    10. Lee, Ying-Ying, 2018. "Efficient propensity score regression estimators of multivalued treatment effects for the treated," Journal of Econometrics, Elsevier, vol. 204(2), pages 207-222.
    11. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    12. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    13. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    14. Chung Choe & Alfonso Flores-Lagunes & Sang-Jun Lee, 2015. "Do dropouts with longer training exposure benefit from training programs? Korean evidence employing methods for continuous treatments," Empirical Economics, Springer, vol. 48(2), pages 849-881, March.
    15. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    16. Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2017. "The finite sample performance of semi- and non-parametric estimators for treatment effects and policy evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 91-102.
    17. Martin Biewen & Bernd Fitzenberger & Aderonke Osikominu & Marie Paul, 2014. "The Effectiveness of Public-Sponsored Training Revisited: The Importance of Data and Methodological Choices," Journal of Labor Economics, University of Chicago Press, vol. 32(4), pages 837-897.
    18. Ying-Ying Lee, 2015. "Efficient propensity score regression estimators of multi-valued treatment effects for the treated," Economics Series Working Papers 738, University of Oxford, Department of Economics.
    19. Gueorgui Kambourov & Iourii Manovskii & Miana Plesca, 2020. "Occupational mobility and the returns to training," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(1), pages 174-211, February.
    20. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.

    More about this item

    Keywords

    Multiple treatments; Generalized propensity score; Local economic conditions;
    All these keywords.

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:mia:wpaper:2011-10. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Daniela Valdivia (email available below). General contact details of provider: https://edirc.repec.org/data/demiaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.