IDEAS home Printed from https://ideas.repec.org/p/mia/wpaper/2010-9.html
   My bibliography  Save this paper

Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data

Author

Listed:
  • Carlos A. Flores

    () (Department of Economics, University of Miami)

  • Oscar A. Mitnik

    () (Department of Economics, University of Miami and IZA)

Abstract

This paper assesses the e¤ectiveness of unconfoundedness-based estimators of mean e¤ects for multiple or multivalued treatments in eliminating biases arising from nonrandom treatment assignment. We evaluate these multiple treatment estimators by simultaneously equalizing average outcomes among several control groups from a randomized experiment. We study linear regression estimators as well as partial mean and weighting estimators based on the generalized propensity score (GPS). We also study the use of the GPS in assessing the comparability of individuals among the di¤erent treatment groups, and propose a strategy to determine the overlap or common support region that is less stringent than those previously used in the literature. Our results show that in the multiple treatment setting there may be treatment groups for which it is extremely di¢ cult to ?nd valid comparison groups, and that the GPS plays a signi?cant role in identifying those groups. In such situations, the estimators we consider perform poorly. However, their performance improves considerably once attention is restricted to those treatment groups with adequate overlap quality, with di¤erence-in-di¤erence estimators performing the best. Our results suggest that unconfoundedness-based estimators are a valuable econometric tool for evaluating multiple treatments, as long as the overlap quality is satisfactory.

Suggested Citation

  • Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-9, University of Miami, Department of Economics.
  • Handle: RePEc:mia:wpaper:2010-9
    as

    Download full text from publisher

    File URL: http://moya.bus.miami.edu/~omitnik/PDF_Documents/Evaluating_Mult_Treatments.pdf
    File Function: First version, 2009
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2009. "Dealing with limited overlap in estimation of average treatment effects," Biometrika, Biometrika Trust, vol. 96(1), pages 187-199.
    2. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82.
    3. Robert J. LaLonde, 1984. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," Working Papers 563, Princeton University, Department of Economics, Industrial Relations Section..
    4. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, pages 205-220.
    5. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the California GAIN Program," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 521-566, July.
    6. Markus Frölich & Almas Heshmati & Michael Lechner, 2004. "A microeconometric evaluation of rehabilitation of long-term sickness in Sweden," Journal of Applied Econometrics, John Wiley & Sons, Ltd., pages 375-396.
    7. James J. Heckman & Jeffrey A. Smith, 2004. "The Determinants of Participation in a Social Program: Evidence from a Prototypical Job Training Program," Journal of Labor Economics, University of Chicago Press, vol. 22(2), pages 243-298, April.
    8. Markus Fr–lich, 2004. "Programme Evaluation with Multiple Treatments," Journal of Economic Surveys, Wiley Blackwell, pages 181-224.
    9. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
    10. David S. Lee & Thomas Lemieux, 2009. "Regression Discontinuity Designs In Economics," Working Papers 1118, Princeton University, Department of Economics, Industrial Relations Section..
    11. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 605-654.
    12. Jere R. Behrman & Yingmei Cheng & Petra E. Todd, 2004. "Evaluating Preschool Programs When Length of Exposure to the Program Varies: A Nonparametric Approach," The Review of Economics and Statistics, MIT Press, pages 108-132.
    13. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, pages 281-355.
    14. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, pages 305-353.
    15. Chamberlain, Gary, 1986. "Asymptotic efficiency in semi-parametric models with censoring," Journal of Econometrics, Elsevier, pages 189-218.
    16. Kluve, Jochen & Schneider, Hilmar & Uhlendorff, Arne & Zhao, Zhong, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Ruhr Economic Papers 35, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    17. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, pages 138-154.
    18. Black, Dan A. & Smith, J.A.Jeffrey A., 2004. "How robust is the evidence on the effects of college quality? Evidence from matching," Journal of Econometrics, Elsevier, pages 99-124.
    19. Miana Plesca & Jeffrey Smith, 2007. "Evaluating multi-treatment programs: theory and evidence from the U.S. Job Training Partnership Act experiment," Empirical Economics, Springer, pages 491-528.
    20. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, pages 761-783.
    21. Markus Frölich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, pages 77-90.
    22. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, pages 923-937.
    23. Sydney Ludvigson & Christina H. Paxson, 2001. "Approximation Bias In Linearized Euler Equations," The Review of Economics and Statistics, MIT Press, pages 242-256.
    24. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    25. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, pages 1281-1301.
    26. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, pages 5-86.
    27. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, pages 1161-1189.
    28. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute for the Study of Labor (IZA).
    29. Kluve, Jochen & Schneider, Hilmar & Uhlendorff, Arne & Zhao, Zhong, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Ruhr Economic Papers 35, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    30. Horowitz, Joel L. & Lee, Sokbae, 2005. "Nonparametric Estimation of an Additive Quantile Regression Model," Journal of the American Statistical Association, American Statistical Association, pages 1238-1249.
    31. V. Joseph Hotz & Guido W. Imbens & Julie H. Mortimer, 1999. "Predicting the Efficacy of Future Training Programs Using Past Experiences," NBER Technical Working Papers 0238, National Bureau of Economic Research, Inc.
    32. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    33. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, pages 151-161.
    34. Oscar A. Mitnik, 2007. "Intergenerational transmission of welfare dependency: The effects of length of exposure," Working Papers 0715, University of Miami, Department of Economics.
    35. Carlos A. Flores, 2007. "Estimation of Dose-Response Functions and Optimal Doses with a Continuous Treatment," Working Papers 0707, University of Miami, Department of Economics.
    36. Newey, Whitney K., 1994. "Kernel Estimation of Partial Means and a General Variance Estimator," Econometric Theory, Cambridge University Press, vol. 10(02), pages 1-21, June.
    37. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, pages 5-86.
    38. Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," Working Papers 1042, Princeton University, Department of Economics, Industrial Relations Section..
    39. Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2007. "Estimating the Effects of Length of Exposure to a Training Program: The Case of Job Corps," Working Papers 1042, Princeton University, Department of Economics, Industrial Relations Section..
    40. Kluve, Jochen & Schneider, Hilmar & Uhlendorff, Arne & Zhao, Zhong, 2007. "Evaluating continuous training programs using the generalized propensity score1," Technical Reports 2007,39, Technische Universität Dortmund, Sonderforschungsbereich 475: Komplexitätsreduktion in multivariaten Datenstrukturen.
    41. Jeremy Lise & Shannon Seitz & Jeffrey Smith, 2003. "Equilibrium Policy Experiments and the Evaluation of Social Programs," Working Papers 1012, Queen's University, Department of Economics.
    42. Charles Michalopoulos & Howard S. Bloom & Carolyn J. Hill, 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs?," The Review of Economics and Statistics, MIT Press, pages 156-179.
    43. Dehejia, Rajeev, 2005. "Practical propensity score matching: a reply to Smith and Todd," Journal of Econometrics, Elsevier, pages 355-364.
    44. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, pages 4-29.
    45. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    46. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    47. Heckman, James J, 1990. "Varieties of Selection Bias," American Economic Review, American Economic Association, pages 313-318.
    48. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, pages 604-620.
    49. repec:zbw:rwirep:0035 is not listed on IDEAS
    50. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, pages 91-107.
    51. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics, Palgrave Macmillan.
    52. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    53. Thomas Fraker & Rebecca Maynard, 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs," Journal of Human Resources, University of Wisconsin Press, pages 194-227.
    54. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, pages 241-270.
    55. Manning, Willard G. & Basu, Anirban & Mullahy, John, 2005. "Generalized modeling approaches to risk adjustment of skewed outcomes data," Journal of Health Economics, Elsevier, pages 465-488.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. UCHINO Taisuke & UESUGI Iichiro, 2012. "The Effects of a Megabank Merger on Firm-Bank Relationships and Borrowing Costs," Discussion papers 12022, Research Institute of Economy, Trade and Industry (RIETI).
    2. Bertrand, Olivier & Betschinger, Marie-Ann, 2012. "Performance of domestic and cross-border acquisitions: Empirical evidence from Russian acquirers," Journal of Comparative Economics, Elsevier, pages 413-437.
    3. Martin Huber & Michael Lechner & Conny Wunsch, 2010. "How to control for many covariates? Reliable estimators based on the propensity score," University of St. Gallen Department of Economics working paper series 2010 2010-30, Department of Economics, University of St. Gallen.
    4. Rafael Perez Ribas & Fabio Veras Soares & Clarissa Gondim Teixeira & Elydia Silva & Guilherme Issamu Hirata, 2010. "Beyond Cash: Assessing Externality and Behaviour Effects of Non-Experimental Cash Transfers," Working Papers 65, International Policy Centre for Inclusive Growth.
    5. Choe, Chung & Flores-Lagunes, Alfonso & Lee, Sang-Jun, 2011. "Do Dropouts Benefit from Training Programs? Korean Evidence Employing Methods for Continuous Treatments," IZA Discussion Papers 6064, Institute for the Study of Labor (IZA).
    6. Martin Huber & Michael Lechner & Conny Wunsch, 2010. "How to control for many covariates? Reliable estimators based on the propensity score," University of St. Gallen Department of Economics working paper series 2010 2010-30, Department of Economics, University of St. Gallen.
    7. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, pages 1-21.
    8. Rodney J. Andrews & Jing Li & Michael F. Lovenheim, 2012. "Quantile Treatment Effects of College Quality on Earnings: Evidence from Administrative Data in Texas," NBER Working Papers 18068, National Bureau of Economic Research, Inc.
    9. Taisuke Uchino & Iichiro Uesugi, 2012. "The Effects of a Megabank Merger on Firm-Bank Relationships and Borrowing Costs," Global COE Hi-Stat Discussion Paper Series gd12-233, Institute of Economic Research, Hitotsubashi University.

    More about this item

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:mia:wpaper:2010-9. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher Parmeter). General contact details of provider: http://edirc.repec.org/data/demiaus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.