Advanced Search
MyIDEAS: Login

How to control for many covariates? Reliable estimators based on the propensity score

Contents:

Author Info

  • Martin Huber

    ()

  • Michael Lechner

    ()

  • Conny Wunsch

    ()

Abstract

We investigate the finite sample properties of a large number of estimators for the average treatment effect on the treated that are suitable when adjustment for observable covariates is required, like inverse pro¬bability weighting, kernel and other variants of matching, as well as different parametric models. The simulation design used is based on real data usually employed for the evaluation of labour market programmes in Germany. We vary several dimensions of the design that are of practical importance, like sample size, the type of the outcome variable, and aspects of the selection process. We find that trimming individual observations with too much weight as well as the choice of tuning parameters is important for all estimators. The key conclusion from our simulations is that a particular radius matching estimator combined with regression performs best overall, in particular when robustness to misspecifications of the propensity score is considered an important property.

Download Info

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
File URL: http://www1.vwa.unisg.ch/RePEc/usg/dp2010/DP-1030-Le.pdf
Download Restriction: no

Bibliographic Info

Paper provided by Department of Economics, University of St. Gallen in its series University of St. Gallen Department of Economics working paper series 2010 with number 2010-30.

as in new window
Length: 62 pages
Date of creation: Oct 2010
Date of revision:
Handle: RePEc:usg:dp2010:2010-30

Contact details of provider:
Postal: Dufourstrasse 50, CH - 9000 St.Gallen
Phone: +41 71 224 23 25
Fax: +41 71 224 31 35
Email:
Web page: http://www.seps.unisg.ch/
More information through EDIRC

Related research

Keywords: Propensity score matching; kernel matching; inverse probability weighting; selection on observables; empirical Monte Carlo study; finite sample properties;

Other versions of this item:

Find related papers by JEL classification:

This paper has been announced in the following NEP Reports:

References

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
as in new window
  1. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2004. "Dealing with Limited Overlap in Estimation of Average Treatment Effects," Working Papers 0716, University of Miami, Department of Economics, revised 12 Jun 2007.
  2. Guido W. Imbens & Whitney Newey & Geert Ridder, 2006. "Mean-squared-error Calculations for Average Treatment Effects," IEPR Working Papers 06.57, Institute of Economic Policy Research (IEPR).
  3. Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-9, University of Miami, Department of Economics.
  4. Stefanie Behncke & Markus Frölich & Michael Lechner, 2010. "A Caseworker Like Me - Does The Similarity Between The Unemployed and Their Caseworkers Increase Job Placements?," Economic Journal, Royal Economic Society, vol. 120(549), pages 1430-1459, December.
  5. Marco Caliendo & Reinhard Hujer & Stephan L. Thomsen, 2006. "Sectoral Heterogeneity in the Employment Effects of Job Creation Schemes in Germany," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), Justus-Liebig University Giessen, Department of Statistics and Economics, vol. 226(2), pages 139-179, March.
  6. Conny Wunsch & Michael Lechner, 2007. "What Did All the Money Do? On the General Ineffectiveness of Recent West German Labour Market Programmes," University of St. Gallen Department of Economics working paper series 2007 2007-19, Department of Economics, University of St. Gallen.
  7. Jose C. GALDO & Jeffrey SMITH & Dan BLACK, 2008. "Bandwidth Selection and the Estimation of Treatment Effects with Unbalanced Data," Annales d'Economie et de Statistique, ENSAE, issue 91-92, pages 189-216.
  8. Frölich, Markus, 2002. "Nonparametric IV Estimation of Local Average Treatment Effects with Covariates," IZA Discussion Papers 588, Institute for the Study of Labor (IZA).
  9. Sokbae 'Simon' Lee & Yoon-Jae Whang, 2009. "Nonparametric tests of conditional treatment effects," CeMMAP working papers CWP36/09, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  10. Marco Caliendo & Reinhard Hujer & Stephan Thomsen, 2008. "Identifying effect heterogeneity to improve the efficiency of job creation schemes in Germany," Applied Economics, Taylor and Francis Journals, vol. 40(9), pages 1101-1122.
  11. Imbens, Guido & Wooldridge, Jeffrey M., 2009. "Recent Developments in the Econometrics of Program Evaluation," Scholarly Articles 3043416, Harvard University Department of Economics.
  12. Martin Huber, 2011. "Testing for covariate balance using quantile regression and resampling methods," Journal of Applied Statistics, Taylor and Francis Journals, vol. 38(12), pages 2881-2899, February.
  13. Lechner, Michael, 1996. "An Evaluation of Public Sector Sponsored Continuous Vocational Training Programs in East Germany," Discussion Papers 539, Institut fuer Volkswirtschaftslehre und Statistik, Abteilung fuer Volkswirtschaftslehre.
  14. Jeffrey Smith & Petra Todd, 2003. "Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators?," University of Western Ontario, CIBC Centre for Human Capital and Productivity Working Papers 20035, University of Western Ontario, CIBC Centre for Human Capital and Productivity.
  15. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492 National Bureau of Economic Research, Inc.
  16. Lechner, Michael & Miquel, Ruth & Wunsch, Conny, 2004. "Long-Run Effects of Public Sector Sponsored Training in West Germany," IZA Discussion Papers 1443, Institute for the Study of Labor (IZA).
  17. Caliendo, Marco & Hujer, Reinhard & Thomsen, Stephan Lothar, 2005. "The Employment Effects of Job Creation Schemes in Germany: A Microeconometric Evaluation," IZA Discussion Papers 1512, Institute for the Study of Labor (IZA).
  18. Peter Hall & Jeff Racine & Qi Li, 2004. "Cross-Validation and the Estimation of Conditional Probability Densities," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 1015-1026, December.
  19. Gerfin, Michael & Lechner, Michael, 2000. "Microeconometric Evaluation of the Active Labour Market Policy in Switzerland," IZA Discussion Papers 154, Institute for the Study of Labor (IZA).
  20. Daniel Millimet & Rusty Tchernis, 2008. "On the Specification of Propensity Scores: with Applications to the Analysis of Trade Policies," Caepr Working Papers 2006-013_Updated, Center for Applied Economics and Policy Research, Economics Department, Indiana University Bloomington.
  21. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2009. "Does Leaving Welfare Improve Health? Evidence for Germany," IZA Discussion Papers 4370, Institute for the Study of Labor (IZA).
  22. Hujer, Reinhard & Caliendo, Marco & Thomsen, Stephan Lothar, 2003. "New Evidence on the Effects of Job Creation Schemes in Germany - A Matching Approach with Threefold Heterogeneity," IZA Discussion Papers 750, Institute for the Study of Labor (IZA).
  23. Lechner, Michael & Wunsch, Conny, 2007. "Are training programs more effective when unemployment is high?," IAB Discussion Paper 200707, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
  24. Blundell, Richard William & Costa Dias, Monica & Meghir, Costas & Van Reenen, John, 2003. "Evaluating the Employment Impact of a Mandatory Job Search Programme," CEPR Discussion Papers 3786, C.E.P.R. Discussion Papers.
  25. Stefanie Behncke & Markus Frˆlich & Michael Lechner, 2010. "Unemployed and their caseworkers: should they be friends or foes?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(1), pages 67-92.
  26. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2000. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," NBER Technical Working Papers 0251, National Bureau of Economic Research, Inc.
  27. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
  28. Guido M. Imbens & Jeffrey M. Wooldridge, 2008. "Recent Developments in the Econometrics of Program Evaluation," NBER Working Papers 14251, National Bureau of Economic Research, Inc.
  29. Augurzky, Boris & Kluve, Jochen, 2004. "Assessing the Performance of Matching Algorithms When Selection into Treatment Is Strong," IZA Discussion Papers 1301, Institute for the Study of Labor (IZA).
  30. Dehejia, Rajeev, 2005. "Practical propensity score matching: a reply to Smith and Todd," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 355-364.
  31. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
  32. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
  33. Joshua Angrist & Jinyong Hahn, 2004. "When to Control for Covariates? Panel Asymptotics for Estimates of Treatment Effects," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 58-72, February.
  34. Michael Lechner & Conny Wunsch, 2009. "Active labour market policy in East Germany," The Economics of Transition, The European Bank for Reconstruction and Development, vol. 17(4), pages 661-702, October.
  35. Alberto Abadie & Guido W. Imbens, 2009. "Matching on the Estimated Propensity Score," NBER Working Papers 15301, National Bureau of Economic Research, Inc.
  36. Ahmed Khwaja & Gabriel Picone & Martin Salm & Justin G. Trogdon, 2011. "A comparison of treatment effects estimators using a structural model of AMI treatment choices and severity of illness information from hospital charts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(5), pages 825-853, 08.
  37. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-20, September.
  38. Michael Lechner, 2010. "The Estimation of Causal Effects by Difference-in-Difference Methods," University of St. Gallen Department of Economics working paper series 2010 2010-28, Department of Economics, University of St. Gallen, revised Oct 2011.
  39. Newey, Whitney K., 1984. "A method of moments interpretation of sequential estimators," Economics Letters, Elsevier, vol. 14(2-3), pages 201-206.
  40. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
  41. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 91-107, February.
  42. Busso, Matias & DiNardo, John & McCrary, Justin, 2009. "New Evidence on the Finite Sample Properties of Propensity Score Matching and Reweighting Estimators," IZA Discussion Papers 3998, Institute for the Study of Labor (IZA).
  43. Heckman, James J & Ichimura, Hidehiko & Todd, Petra, 1998. "Matching as an Econometric Evaluation Estimator," Review of Economic Studies, Wiley Blackwell, vol. 65(2), pages 261-94, April.
  44. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, 01.
  45. Lechner, Michael, 2009. "Long-run labour market and health effects of individual sports activities," Journal of Health Economics, Elsevier, vol. 28(4), pages 839-854, July.
  46. Reinhard Hujer & Stephan Thomsen & Christopher Zeiss, 2006. "The effects of vocational training programmes on the duration of unemployment in Eastern Germany," AStA Advances in Statistical Analysis, Springer, vol. 90(2), pages 299-321, June.
  47. Hujer, Reinhard & Thomsen, Stephan L., 2010. "How do the employment effects of job creation schemes differ with respect to the foregoing unemployment duration?," Labour Economics, Elsevier, vol. 17(1), pages 38-51, January.
  48. Zhao, Zhong, 2006. "Matching Estimators and the Data from the National Supported Work Demonstration Again," IZA Discussion Papers 2375, Institute for the Study of Labor (IZA).
Full references (including those not matched with items on IDEAS)

Citations

Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
as in new window

Cited by:
  1. Martin Biewen & Bernd Fitzenberger & Aderonke Osikominu & Marie Paul, 2012. "The effectiveness of public sponsored training revisited: The importance of data and methodological choices," ECON - Working Papers 091, Department of Economics - University of Zurich.
  2. Van Biesebroeck, Johannes & Yu, Emily & Chen, Shenjie, 2010. "The impact of trade promotion services on Canadian exporter performance," Open Access publications from Katholieke Universiteit Leuven urn:hdl:123456789/267252, Katholieke Universiteit Leuven.
  3. Dolton, Peter & Smith, Jeffrey A., 2011. "The Impact of the UK New Deal for Lone Parents on Benefit Receipt," IZA Discussion Papers 5491, Institute for the Study of Labor (IZA).
  4. Lechner, Michael & Wunsch, Conny, 2011. "Sensitivity of Matching-Based Program Evaluations to the Availability of Control Variables," IZA Discussion Papers 5553, Institute for the Study of Labor (IZA).
  5. Caliendo, Marco & Künn, Steffen & Schmidl, Ricarda, 2011. "Fighting Youth Unemployment: The Effects of Active Labor Market Policies," IZA Discussion Papers 6222, Institute for the Study of Labor (IZA).
  6. Millimet, Daniel L. & Roy, Jayjit, 2011. "Three New Empirical Tests of the Pollution Haven Hypothesis When Environmental Regulation is Endogenous," IZA Discussion Papers 5911, Institute for the Study of Labor (IZA).
  7. Mario Liebensteiner, 2012. "Estimating the Income Gain of Seasonal Labour Migration," WIFO Working Papers 430, WIFO.
  8. Huber, Martin, 2012. "Identifying causal mechanisms in experiments (primarily) based on inverse probability weighting," Economics Working Paper Series 1213, University of St. Gallen, School of Economics and Political Science.
  9. Bühler, Stefan & Helm, Marco & Lechner, Michael, 2011. "Trade Liberalization and Growth: Plant-Level Evidence from Switzerland," Economics Working Paper Series 1133, University of St. Gallen, School of Economics and Political Science.

Lists

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

Statistics

Access and download statistics

Corrections

When requesting a correction, please mention this item's handle: RePEc:usg:dp2010:2010-30

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Martina Flockerzi).

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.