IDEAS home Printed from https://ideas.repec.org/p/fri/fribow/fribow00454.html
   My bibliography  Save this paper

The finite sample performance of semi- and nonparametric estimators for treatment effects and policy evaluation

Author

Listed:
  • Frölich, Markus
  • Huber, Martin
  • Wiesenfarth, Manuel

Abstract

This paper investigates the finite sample performance of a comprehensive set of semi- and nonparametric estimators for treatment and policy evaluation. In contrast to previous simulation studies which mostly considered semiparametric approaches relying on parametric propensity score estimation, we also consider more flexible approaches based on semi- or nonparametric propensity scores, nonparametric regression, and direct covariate matching. In addition to (pair, radius, and kernel) matching, inverse probability weighting, regression, and doubly robust estimation, our studies also cover recently proposed estimators such as genetic matching, entropy balancing, and empirical likelihood estimation. We vary a range of features (sample size, selection into treatment, effect heterogeneity, and correct/misspecification) in our simulations and find that several nonparametric estimators by and large outperform commonly used treatment estimators using a parametric propensity score. Nonparametric regression, nonparametric doubly robust estimation, nonparametric IPW, and one-to-many covariate matching perform best.

Suggested Citation

  • Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2015. "The finite sample performance of semi- and nonparametric estimators for treatment effects and policy evaluation," FSES Working Papers 454, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
  • Handle: RePEc:fri:fribow:fribow00454
    as

    Download full text from publisher

    File URL: http://doc.rero.ch/record/234687/files/WP_SES_454.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Stefanie Behncke & Markus Frölich & Michael Lechner, 2010. "A Caseworker Like Me - Does The Similarity Between The Unemployed and Their Caseworkers Increase Job Placements?," Economic Journal, Royal Economic Society, vol. 120(549), pages 1430-1459, December.
    2. Hugo Ñopo, 2008. "Matching as a Tool to Decompose Wage Gaps," The Review of Economics and Statistics, MIT Press, vol. 90(2), pages 290-299, May.
    3. Michael Lechner & Ruth Miquel & Conny Wunsch, 2011. "Long‐Run Effects Of Public Sector Sponsored Training In West Germany," Journal of the European Economic Association, European Economic Association, vol. 9(4), pages 742-784, August.
    4. Hainmueller, Jens, 2012. "Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies," Political Analysis, Cambridge University Press, vol. 20(01), pages 25-46, December.
    5. Martin Huber & Michael Lechner & Giovanni Mellace, 2016. "The Finite Sample Performance of Estimators for Mediation Analysis Under Sequential Conditional Independence," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(1), pages 139-160, January.
    6. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
    7. van der Laan Mark J. & Polley Eric C & Hubbard Alan E., 2007. "Super Learner," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 6(1), pages 1-23, September.
    8. Matias Busso & John DiNardo & Justin McCrary, 2014. "New Evidence on the Finite Sample Properties of Propensity Score Reweighting and Matching Estimators," The Review of Economics and Statistics, MIT Press, vol. 96(5), pages 885-897, December.
    9. Bodory, Hugo & Camponovo, Lorenzo & Huber, Martin & Lechner, Michael, 2016. "The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators," IZA Discussion Papers 9706, Institute for the Study of Labor (IZA).
    10. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    11. Stefanie Behncke & Markus Frˆlich & Michael Lechner, 2010. "Unemployed and their caseworkers: should they be friends or foes?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(1), pages 67-92.
    12. Racine, Jeff & Li, Qi, 2004. "Nonparametric estimation of regression functions with both categorical and continuous data," Journal of Econometrics, Elsevier, vol. 119(1), pages 99-130, March.
    13. van der Laan Mark J. & Rubin Daniel, 2006. "Targeted Maximum Likelihood Learning," The International Journal of Biostatistics, De Gruyter, vol. 2(1), pages 1-40, December.
    14. Bryan S. Graham & Cristine Campos de Xavier Pinto & Daniel Egel, 2016. "Efficient Estimation of Data Combination Models by the Method of Auxiliary-to-Study Tilting (AST)," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(2), pages 288-301, April.
    15. Bryan S. Graham & Cristine Campos De Xavier Pinto & Daniel Egel, 2012. "Inverse Probability Tilting for Moment Condition Models with Missing Data," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 1053-1079.
    16. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    17. Winfried Pohlmeier & Ruben R. Seiberlich & S. Derya Uysal, 2013. "A Simple and Successul Method to Shrink the Weight," Working Paper Series of the Department of Economics, University of Konstanz 2013-05, Department of Economics, University of Konstanz.
    18. Gruber Susan & van der Laan Mark J., 2010. "A Targeted Maximum Likelihood Estimator of a Causal Effect on a Bounded Continuous Outcome," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-18, August.
    19. Rothe, Christoph & Firpo, Sergio Pinheiro, 2013. "Semiparametric estimation and inference using doubly robust moment conditions," Textos para discussão 330, FGV/EESP - Escola de Economia de São Paulo, Getulio Vargas Foundation (Brazil).
    20. Abadie, Alberto & Imbens, Guido W., 2011. "Bias-Corrected Matching Estimators for Average Treatment Effects," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(1), pages 1-11.
    21. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    22. Hayfield, Tristen & Racine, Jeffrey S., 2008. "Nonparametric Econometrics: The np Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 27(i05).
    23. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    24. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    25. repec:tpr:restat:v:99:y:2017:i:1:p:180-183 is not listed on IDEAS
    26. Alexis Diamond & Jasjeet S. Sekhon, 2013. "Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies," The Review of Economics and Statistics, MIT Press, vol. 95(3), pages 932-945, July.
    27. Li, Qi & Racine, Jeffrey S. & Wooldridge, Jeffrey M., 2009. "Efficient Estimation of Average Treatment Effects with Mixed Categorical and Continuous Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(2), pages 206-223.
    28. Sekhon, Jasjeet S., 2011. "Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching package for R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i07).
    29. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    30. Lechner, Michael & Strittmatter, Anthony, 2014. "Practical Procedures to Deal with Common Support Problems in Matching Estimation," Economics Working Paper Series 1410, University of St. Gallen, School of Economics and Political Science.
    31. Zhao, Zhong, 2008. "Sensitivity of propensity score methods to the specifications," Economics Letters, Elsevier, vol. 98(3), pages 309-319, March.
    32. van der Laan Mark & Gruber Susan, 2016. "One-Step Targeted Minimum Loss-based Estimation Based on Universal Least Favorable One-Dimensional Submodels," The International Journal of Biostatistics, De Gruyter, vol. 12(1), pages 351-378, May.
    33. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    34. Martin Huber & Michael Lechner & Andreas Steinmayr, 2015. "Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour," Empirical Economics, Springer, vol. 49(1), pages 1-31, August.
    35. Klein, Roger W & Spady, Richard H, 1993. "An Efficient Semiparametric Estimator for Binary Response Models," Econometrica, Econometric Society, vol. 61(2), pages 387-421, March.
    36. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    37. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," Review of Economic Studies, Oxford University Press, vol. 65(2), pages 261-294.
    38. Zhong Zhao, 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 91-107, February.
    39. Martin Huber & Michael Lechner & Giovanni Mellace, 2017. "Why Do Tougher Caseworkers Increase Employment? The Role of Program Assignment as a Causal Mechanism," The Review of Economics and Statistics, MIT Press, vol. 99(1), pages 180-183, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andrea ALBANESE & Bart COCKX, 2015. "Permanent Wage Cost Subsidies for Older Workers. An Effective Tool for Increasing Working Time and Postponing Early Retirement?," Discussion Papers (IRES - Institut de Recherches Economiques et Sociales) 2015006, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    2. Bryson, Alex & White, Michael, 2017. "HRM and Small-Firm Employee Motivation: Before and after the Recession," IZA Discussion Papers 10737, Institute for the Study of Labor (IZA).
    3. Andrea Albanese & Bart Cockx & Yannick Thuy, 2015. "Working Time Reductions at the End of the Career. Do they prolong the Time Spent in Employment?," Discussion Papers (IRES - Institut de Recherches Economiques et Sociales) 2015024, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    4. Andrea Albanese & Lorenzo Cappellari & Marco Leonardi, 2017. "The Effects of Youth Labor Market Reforms: Evidence from Italian Apprenticeships," DISCE - Working Papers del Dipartimento di Economia e Finanza def057, Università Cattolica del Sacro Cuore, Dipartimenti e Istituti di Scienze Economiche (DISCE).
    5. repec:eee:econom:v:204:y:2018:i:2:p:207-222 is not listed on IDEAS

    More about this item

    Keywords

    treatment effects; policy evaluation; simulation; empirical Monte Carlo study; propensity score; semi- and nonparametric estimation;

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fri:fribow:fribow00454. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Ivo raemy). General contact details of provider: http://edirc.repec.org/data/wsffrch.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.