IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data

  • Flores, Carlos A.

    ()

    (University of Miami)

  • Mitnik, Oscar A.

    ()

    (University of Miami)

This paper assesses the effectiveness of unconfoundedness-based estimators of mean effects for multiple or multivalued treatments in eliminating biases arising from nonrandom treatment assignment. We evaluate these multiple treatment estimators by simultaneously equalizing average outcomes among several control groups from a randomized experiment. We study linear regression estimators as well as partial mean and weighting estimators based on the generalized propensity score (GPS). We also study the use of the GPS in assessing the comparability of individuals among the different treatment groups, and propose a strategy to determine the overlap or common support region that is less stringent than those previously used in the literature. Our results show that in the multiple treatment setting there may be treatment groups for which it is extremely difficult to find valid comparison groups, and that the GPS plays a significant role in identifying those groups. In such situations, the estimators we consider perform poorly. However, their performance improves considerably once attention is restricted to those treatment groups with adequate overlap quality, with difference-in-difference estimators performing the best. Our results suggest that unconfoundedness-based estimators are a valuable econometric tool for evaluating multiple treatments, as long as the overlap quality is satisfactory.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://ftp.iza.org/dp4451.pdf
Download Restriction: no

Paper provided by Institute for the Study of Labor (IZA) in its series IZA Discussion Papers with number 4451.

as
in new window

Length: 53 pages
Date of creation: Sep 2009
Date of revision:
Handle: RePEc:iza:izadps:dp4451
Contact details of provider: Postal: IZA, P.O. Box 7240, D-53072 Bonn, Germany
Phone: +49 228 3894 223
Fax: +49 228 3894 180
Web page: http://www.iza.org

Order Information: Postal: IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany
Email:


References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Guido Imbens, 2000. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometric Society World Congress 2000 Contributed Papers 1166, Econometric Society.
  2. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Discussion Papers of DIW Berlin 752, DIW Berlin, German Institute for Economic Research.
  3. Dan A. Black & Jeffrey Smith, 2003. "How Robust is the Evidence on the Effects of College Quality? Evidence From Matching," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20033, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
  4. Guido W. Imbens, 1999. "The Role of the Propensity Score in Estimating Dose-Response Functions," NBER Technical Working Papers 0237, National Bureau of Economic Research, Inc.
  5. Guido M. Imbens & Jeffrey M. Wooldridge, 2008. "Recent Developments in the Econometrics of Program Evaluation," NBER Working Papers 14251, National Bureau of Economic Research, Inc.
  6. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
  7. David S. Lee & Thomas Lemieux, 2009. "Regression Discontinuity Designs In Economics," Working Papers 1118, Princeton University, Department of Economics, Industrial Relations Section..
  8. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Re-Analysis of the California GAIN Program," NBER Working Papers 11939, National Bureau of Economic Research, Inc.
  9. Guido W. Imbens, 2003. "Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review," NBER Technical Working Papers 0294, National Bureau of Economic Research, Inc.
  10. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity score matching methods for non-experimental causal studies," Discussion Papers 0102-14, Columbia University, Department of Economics.
  11. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82.
  12. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-37, September.
  13. Newey, Whitney K., 1994. "Kernel Estimation of Partial Means and a General Variance Estimator," Econometric Theory, Cambridge University Press, vol. 10(02), pages 1-21, June.
  14. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
  15. Jere R. Behrman & Yingmei Cheng & Petra E. Todd, 2004. "Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 108-132, February.
  16. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
  17. V. Joseph Hotz & Guido W. Imbens & Julie H. Mortimer, 1999. "Predicting the Efficacy of Future Training Programs Using Past Experiences," NBER Technical Working Papers 0238, National Bureau of Economic Research, Inc.
  18. Jeffrey M. Wooldridge, 2004. "Inverse probability weighted estimation for general missing data problems," CeMMAP working papers CWP05/04, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  19. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 156-179, February.
  20. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
  21. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute for the Study of Labor (IZA).
  22. Heckman, James J & Ichimura, Hidehiko & Todd, Petra E, 1997. "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Wiley Blackwell, vol. 64(4), pages 605-54, October.
  23. Markus Froelich, 2002. "Programme Evaluation with Multiple Treatments," University of St. Gallen Department of Economics working paper series 2002 2002-17, Department of Economics, University of St. Gallen.
  24. Thomas Fraker & Rebecca Maynard, 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs," Journal of Human Resources, University of Wisconsin Press, vol. 22(2), pages 194-227.
  25. Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
  26. Carlos A. Flores, 2007. "Estimation of Dose-Response Functions and Optimal Doses with a Continuous Treatment," Working Papers 0707, University of Miami, Department of Economics.
  27. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2004. "Dealing with Limited Overlap in Estimation of Average Treatment Effects," Working Papers 0716, University of Miami, Department of Economics, revised 12 Jun 2007.
  28. Frölich, Markus & Heshmati, Almas & Lechner, Michael, 2000. "A Microeconometric Evaluation of Rehabilitation of Long-term Sickness in Sweden," SSE/EFI Working Paper Series in Economics and Finance 373, Stockholm School of Economics, revised 03 Apr 2000.
  29. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  30. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
  31. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics, Palgrave Macmillan.
  32. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
  33. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
  34. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
  35. Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," Working Papers 1042, Princeton University, Department of Economics, Industrial Relations Section..
  36. Chamberlain, Gary, 1986. "Asymptotic efficiency in semi-parametric models with censoring," Journal of Econometrics, Elsevier, vol. 32(2), pages 189-218, July.
  37. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
  38. Dehejia, Rajeev, 2005. "Practical propensity score matching: a reply to Smith and Todd," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 355-364.
  39. Markus Frölich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
  40. Oscar A. Mitnik, 2007. "Intergenerational transmission of welfare dependency: The effects of length of exposure," Working Papers 0715, University of Miami, Department of Economics.
  41. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 91-107, February.
  42. Miana Plesca & Jeffrey Smith, 2007. "Evaluating multi-treatment programs: theory and evidence from the U.S. Job Training Partnership Act experiment," Empirical Economics, Springer, vol. 32(2), pages 491-528, May.
  43. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, 01.
  44. Heckman, James J, 1990. "Varieties of Selection Bias," American Economic Review, American Economic Association, vol. 80(2), pages 313-18, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp4451. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.