Advanced Search
MyIDEAS: Login

Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data

Contents:

Author Info

  • Flores, Carlos A.

    ()
    (University of Miami)

  • Mitnik, Oscar A.

    ()
    (University of Miami)

Abstract

This paper assesses the effectiveness of unconfoundedness-based estimators of mean effects for multiple or multivalued treatments in eliminating biases arising from nonrandom treatment assignment. We evaluate these multiple treatment estimators by simultaneously equalizing average outcomes among several control groups from a randomized experiment. We study linear regression estimators as well as partial mean and weighting estimators based on the generalized propensity score (GPS). We also study the use of the GPS in assessing the comparability of individuals among the different treatment groups, and propose a strategy to determine the overlap or common support region that is less stringent than those previously used in the literature. Our results show that in the multiple treatment setting there may be treatment groups for which it is extremely difficult to find valid comparison groups, and that the GPS plays a significant role in identifying those groups. In such situations, the estimators we consider perform poorly. However, their performance improves considerably once attention is restricted to those treatment groups with adequate overlap quality, with difference-in-difference estimators performing the best. Our results suggest that unconfoundedness-based estimators are a valuable econometric tool for evaluating multiple treatments, as long as the overlap quality is satisfactory.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://ftp.iza.org/dp4451.pdf
Download Restriction: no

Bibliographic Info

Paper provided by Institute for the Study of Labor (IZA) in its series IZA Discussion Papers with number 4451.

as in new window
Length: 53 pages
Date of creation: Sep 2009
Date of revision:
Handle: RePEc:iza:izadps:dp4451

Contact details of provider:
Postal: IZA, P.O. Box 7240, D-53072 Bonn, Germany
Phone: +49 228 3894 223
Fax: +49 228 3894 180
Web page: http://www.iza.org

Order Information:
Postal: IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany
Email:

Related research

Keywords: generalized propensity score; nonexperimental estimators; multiple treatments;

Other versions of this item:

Find related papers by JEL classification:

This paper has been announced in the following NEP Reports:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Markus Frölich & Almas Heshmati & Michael Lechner, 2004. "A microeconometric evaluation of rehabilitation of long-term sickness in Sweden," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(3), pages 375-396.
  2. Thomas Fraker & Rebecca Maynard, 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs," Journal of Human Resources, University of Wisconsin Press, vol. 22(2), pages 194-227.
  3. David S. Lee & Thomas Lemieux, 2009. "Regression Discontinuity Designs in Economics," NBER Working Papers 14723, National Bureau of Economic Research, Inc.
  4. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
  5. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the California GAIN Program," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 521-566, July.
  6. Newey, W.K., 1992. "Kernel Estimation of Partial Means and a General Variance Estimator," Working papers 93-3, Massachusetts Institute of Technology (MIT), Department of Economics.
  7. Guido Imbens, 2000. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometric Society World Congress 2000 Contributed Papers 1166, Econometric Society.
  8. Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," Working Papers 1042, Princeton University, Department of Economics, Industrial Relations Section..
  9. Markus Frölich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
  10. V. Joseph Hotz & Guido W. Imbens & Julie H. Mortimer, 1999. "Predicting the Efficacy of Future Training Programs Using Past Experiences," NBER Technical Working Papers 0238, National Bureau of Economic Research, Inc.
  11. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 91-107, February.
  12. Markus Fr–lich, 2004. "Programme Evaluation with Multiple Treatments," Journal of Economic Surveys, Wiley Blackwell, vol. 18(2), pages 181-224, 04.
  13. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
  14. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
  15. Guido W. Imbens, 1999. "The Role of the Propensity Score in Estimating Dose-Response Functions," NBER Technical Working Papers 0237, National Bureau of Economic Research, Inc.
  16. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Discussion Papers of DIW Berlin 752, DIW Berlin, German Institute for Economic Research.
  17. Black, Dan A. & Smith, J.A.Jeffrey A., 2004. "How robust is the evidence on the effects of college quality? Evidence from matching," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 99-124.
  18. Michael Lechner, 2005. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Labor and Demography 0505006, EconWPA.
  19. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics, Palgrave Macmillan.
  20. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
  21. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," NBER Working Papers 6699, National Bureau of Economic Research, Inc.
  22. Wooldridge, Jeffrey M. & Imbens, Guido, 2009. "Recent Developments in the Econometrics of Program Evaluation," Scholarly Articles 3043416, Harvard University Department of Economics.
  23. Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
  24. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
  25. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
  26. James J. Heckman, 1989. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training," NBER Working Papers 2861, National Bureau of Economic Research, Inc.
  27. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
  28. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2004. "Dealing with Limited Overlap in Estimation of Average Treatment Effects," Working Papers 0716, University of Miami, Department of Economics, revised 12 Jun 2007.
  29. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-37, September.
  30. Jeffrey Smith & Petra Todd, 2003. "Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20035, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
  31. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, 01.
  32. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
  33. Heckman, James J, 1990. "Varieties of Selection Bias," American Economic Review, American Economic Association, vol. 80(2), pages 313-18, May.
  34. Oscar A. Mitnik, 2007. "Intergenerational transmission of welfare dependency: The effects of length of exposure," Working Papers 0715, University of Miami, Department of Economics.
  35. Jere R. Behrman & Yingmei Cheng & Petra E. Todd, 2004. "Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 108-132, February.
  36. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2006. "Using State Administrative Data to Measure Program Performance," Working Papers 0702, Department of Economics, University of Missouri.
  37. Carlos A. Flores, 2007. "Estimation of Dose-Response Functions and Optimal Doses with a Continuous Treatment," Working Papers 0707, University of Miami, Department of Economics.
  38. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  39. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute for the Study of Labor (IZA).
  40. Heckman, James J & Ichimura, Hidehiko & Todd, Petra E, 1997. "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Wiley Blackwell, vol. 64(4), pages 605-54, October.
  41. Miana Plesca & Jeffrey Smith, 2007. "Evaluating multi-treatment programs: theory and evidence from the U.S. Job Training Partnership Act experiment," Empirical Economics, Springer, vol. 32(2), pages 491-528, May.
  42. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 156-179, February.
  43. Chamberlain, Gary, 1986. "Asymptotic efficiency in semi-parametric models with censoring," Journal of Econometrics, Elsevier, vol. 32(2), pages 189-218, July.
  44. Dehejia, Rajeev, 2005. "Practical propensity score matching: a reply to Smith and Todd," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 355-364.
Full references (including those not matched with items on IDEAS)

Citations

Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. UCHINO Taisuke & UESUGI Iichiro, 2012. "The Effects of a Megabank Merger on Firm-Bank Relationships and Borrowing Costs," Discussion papers 12022, Research Institute of Economy, Trade and Industry (RIETI).
  2. Rodney J. Andrews & Jing Li & Michael F. Lovenheim, 2012. "Quantile Treatment Effects of College Quality on Earnings: Evidence from Administrative Data in Texas," NBER Working Papers 18068, National Bureau of Economic Research, Inc.
  3. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
  4. Bertrand, Olivier & Betschinger, Marie-Ann, 2012. "Performance of domestic and cross-border acquisitions: Empirical evidence from Russian acquirers," Journal of Comparative Economics, Elsevier, vol. 40(3), pages 413-437.
  5. Rafael Perez Ribas & Fabio Veras Soares & Clarissa Teixeira & Elydia Silva & Guilherme Hirata, 2011. "Beyond Cash: Assessing Externality and Behaviour Effects of Non-Experimental Cash Transfers," Working Papers PIERI 2011-18, PEP-PIERI.
  6. FLORES-LAGUNES Alfonso & CHOE Chung & LEE Sang-Jun, 2011. "Do Dropouts Benefit from Training Programs? Korean Evidence Employing Methods for Continuous Treatments," CEPS/INSTEAD Working Paper Series 2011-34, CEPS/INSTEAD.
  7. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute for the Study of Labor (IZA).

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp4451. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.