IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data

  • Carlos A. Flores

    ()

    (Department of Economics, University of Miami)

  • Oscar A. Mitnik

    ()

    (Department of Economics, University of Miami and IZA)

This paper assesses the e¤ectiveness of unconfoundedness-based estimators of mean e¤ects for multiple or multivalued treatments in eliminating biases arising from nonrandom treatment assignment. We evaluate these multiple treatment estimators by simultaneously equalizing average outcomes among several control groups from a randomized experiment. We study linear regression estimators as well as partial mean and weighting estimators based on the generalized propensity score (GPS). We also study the use of the GPS in assessing the comparability of individuals among the di¤erent treatment groups, and propose a strategy to determine the overlap or common support region that is less stringent than those previously used in the literature. Our results show that in the multiple treatment setting there may be treatment groups for which it is extremely di¢ cult to ?nd valid comparison groups, and that the GPS plays a signi?cant role in identifying those groups. In such situations, the estimators we consider perform poorly. However, their performance improves considerably once attention is restricted to those treatment groups with adequate overlap quality, with di¤erence-in-di¤erence estimators performing the best. Our results suggest that unconfoundedness-based estimators are a valuable econometric tool for evaluating multiple treatments, as long as the overlap quality is satisfactory.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://moya.bus.miami.edu/~omitnik/PDF_Documents/Evaluating_Mult_Treatments.pdf
File Function: First version, 2009
Download Restriction: no

Paper provided by University of Miami, Department of Economics in its series Working Papers with number 2010-10.

as
in new window

Length: 51 pages
Date of creation: Sep 2009
Date of revision:
Publication status: Published
Handle: RePEc:mia:wpaper:2010-10
Contact details of provider: Postal: P.O. Box 248126, Coral Gables, FL 33124-6550
Phone: (305) 284-5540
Fax: (305) 284-2985
Web page: http://www.bus.miami.edu/faculty-and-research/academic-departments/economics/index.html
More information through EDIRC

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Black, Dan A. & Smith, J.A.Jeffrey A., 2004. "How robust is the evidence on the effects of college quality? Evidence from matching," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 99-124.
  2. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  3. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2009. "Dealing with limited overlap in estimation of average treatment effects," Biometrika, Biometrika Trust, vol. 96(1), pages 187-199.
  4. Frölich, Markus, 2002. "Programme Evaluation with Multiple Treatments," IZA Discussion Papers 542, Institute for the Study of Labor (IZA).
  5. Chamberlain, Gary, 1986. "Asymptotic efficiency in semi-parametric models with censoring," Journal of Econometrics, Elsevier, vol. 32(2), pages 189-218, July.
  6. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2000. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," NBER Technical Working Papers 0251, National Bureau of Economic Research, Inc.
  7. Markus Frölich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
  8. Thomas Fraker & Rebecca Maynard, 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs," Journal of Human Resources, University of Wisconsin Press, vol. 22(2), pages 194-227.
  9. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Discussion Papers of DIW Berlin 752, DIW Berlin, German Institute for Economic Research.
  10. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
  11. Mueser, Peter R. & Troske, Kenneth & Gorislavsky, Alexey, 2003. "Using State Administrative Data to Measure Program Performance," IZA Discussion Papers 786, Institute for the Study of Labor (IZA).
  12. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
  13. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Re-Analysis of the California GAIN Program," NBER Working Papers 11939, National Bureau of Economic Research, Inc.
  14. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
  15. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
  16. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
  17. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
  18. Michael Lechner, 2005. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Labor and Demography 0505006, EconWPA.
  19. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-37, September.
  20. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
  21. Newey, Whitney K., 1994. "Kernel Estimation of Partial Means and a General Variance Estimator," Econometric Theory, Cambridge University Press, vol. 10(02), pages 1-21, June.
  22. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, 01.
  23. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
  24. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics, Palgrave Macmillan.
  25. Oscar A. Mitnik, 2007. "Intergenerational transmission of welfare dependency: The effects of length of exposure," Working Papers 0715, University of Miami, Department of Economics.
  26. Dehejia, Rajeev, 2005. "Practical propensity score matching: a reply to Smith and Todd," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 355-364.
  27. Markus Frölich & Almas Heshmati & Michael Lechner, 2004. "A microeconometric evaluation of rehabilitation of long-term sickness in Sweden," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(3), pages 375-396.
  28. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
  29. Heckman, James J & Ichimura, Hidehiko & Todd, Petra E, 1997. "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Wiley Blackwell, vol. 64(4), pages 605-54, October.
  30. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
  31. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
  32. V. Joseph Hotz & Guido W. Imbens & Julie H. Mortimer, 1999. "Predicting the Efficacy of Future Training Programs Using Past Experiences," NBER Technical Working Papers 0238, National Bureau of Economic Research, Inc.
  33. Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
  34. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
  35. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 91-107, February.
  36. Guido W. Imbens, 1999. "The Role of the Propensity Score in Estimating Dose-Response Functions," NBER Technical Working Papers 0237, National Bureau of Economic Research, Inc.
  37. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
  38. Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," Working Papers 1042, Princeton University, Department of Economics, Industrial Relations Section..
  39. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute for the Study of Labor (IZA).
  40. Carlos A. Flores, 2007. "Estimation of Dose-Response Functions and Optimal Doses with a Continuous Treatment," Working Papers 0707, University of Miami, Department of Economics.
  41. Jere R. Behrman & Yingmei Cheng & Petra E. Todd, 2004. "Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 108-132, February.
  42. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 156-179, February.
  43. Heckman, James J, 1990. "Varieties of Selection Bias," American Economic Review, American Economic Association, vol. 80(2), pages 313-18, May.
  44. Miana Plesca & Jeffrey Smith, 2007. "Evaluating multi-treatment programs: theory and evidence from the U.S. Job Training Partnership Act experiment," Empirical Economics, Springer, vol. 32(2), pages 491-528, May.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:mia:wpaper:2010-10. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher Parmeter)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.