IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/22950.html
   My bibliography  Save this paper

Targeting Policies: Multiple Testing and Distributional Treatment Effects

Author

Listed:
  • Steven F. Lehrer
  • R. Vincent Pohl
  • Kyungchul Song

Abstract

Economic theory often predicts that treatment responses may depend on individuals’ characteristics and location on the outcome distribution. Policymakers need to account for such treatment effect heterogeneity in order to efficiently allocate resources to subgroups that can successfully be targeted by a policy. However, when interpreting treatment effects across subgroups and the outcome distribution, inference has to be adjusted for multiple hypothesis testing to avoid an overestimation of positive treatment effects. We propose six new tests for treatment effect heterogeneity that make corrections for the family-wise error rate and that identify subgroups and ranges of the outcome distribution exhibiting economically and statistically significant treatment effects. We apply these tests to individual responses to welfare reform and show that welfare recipients benefit from the reform in a smaller range of the earnings distribution than previously estimated. Our results shed new light on effectiveness of welfare reform and demonstrate the importance of correcting for multiple testing.

Suggested Citation

  • Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:22950
    Note: LS TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w22950.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    2. Patrick Kline & Melissa Tartari, 2016. "Bounding the Labor Supply Responses to a Randomized Welfare Experiment: A Revealed Preference Approach," American Economic Review, American Economic Association, vol. 106(4), pages 972-1014, April.
    3. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    4. Sergio Firpo, 2007. "Efficient Semiparametric Estimation of Quantile Treatment Effects," Econometrica, Econometric Society, vol. 75(1), pages 259-276, January.
    5. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    6. Maier, Michael, 2011. "Tests for distributional treatment effects under unconfoundedness," Economics Letters, Elsevier, vol. 110(1), pages 49-51, January.
    7. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 487-535.
    8. Emmanuel Saez, 2010. "Do Taxpayers Bunch at Kink Points?," American Economic Journal: Economic Policy, American Economic Association, vol. 2(3), pages 180-212, August.
    9. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    10. Heckman, James J. & Urzúa, Sergio, 2010. "Comparing IV with structural models: What simple IV can and cannot identify," Journal of Econometrics, Elsevier, vol. 156(1), pages 27-37, May.
    11. Michael P. Keane, 2011. "Labor Supply and Taxes: A Survey," Journal of Economic Literature, American Economic Association, vol. 49(4), pages 961-1075, December.
    12. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    13. G�nther Fink & Margaret McConnell & Sebastian Vollmer, 2014. "Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(1), pages 44-57, January.
    14. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    15. Joseph P. Romano & Azeem M. Shaikh, 2010. "Inference for the Identified Set in Partially Identified Econometric Models," Econometrica, Econometric Society, vol. 78(1), pages 169-211, January.
    16. James J. Heckman, 2001. "Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture," Journal of Political Economy, University of Chicago Press, vol. 109(4), pages 673-748, August.
    17. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009 is not listed on IDEAS
    18. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    19. Daniel Friedlander & Philip K. Robins, 1997. "The Distributional Impacts of Social Programs," Evaluation Review, , vol. 21(5), pages 531-553, October.
    20. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    21. repec:pri:cheawb:deaton%20instruments%20of%20development%20keynes%20lecture%202009.pdf is not listed on IDEAS
    22. Abadie A., 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 284-292, March.
    23. Rothe, Christoph, 2010. "Nonparametric estimation of distributional policy effects," Journal of Econometrics, Elsevier, vol. 155(1), pages 56-70, March.
    24. repec:pri:rpdevs:instruments_of_development.pdf is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yu-Chin Hsu & Chung-Ming Kuan & Giorgio Teng-Yu Lo, 2017. "Quantile Treatment Effects in Regression Discontinuity Designs with Covariates," IEAS Working Paper : academic research 17-A009, Institute of Economics, Academia Sinica, Taipei, Taiwan.
    2. Kota Ogasawara & Yukitoshi Matsushita, 2019. "Heterogeneous treatment effects of safe water on infectious disease: Do meteorological factors matter?," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 13(1), pages 55-82, January.
    3. Elena Denisova-Schmidt & Martin Huber & Elvira Leontyeva & Anna Solovyeva, 2021. "Combining experimental evidence with machine learning to assess anti-corruption educational campaigns among Russian university students," Empirical Economics, Springer, vol. 60(4), pages 1661-1684, April.
    4. Steven F. Lehrer & Weili Ding, 2017. "Are genetic markers of interest for economic research?," IZA Journal of Labor Policy, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 6(1), pages 1-23, December.
    5. Arindrajit Dube, 2019. "Minimum Wages and the Distribution of Family Incomes," American Economic Journal: Applied Economics, American Economic Association, vol. 11(4), pages 268-304, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lehrer, Steven F. & Pohl, R. Vincent & Song, Kyungchul, 2018. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," MPRA Paper 89532, University Library of Munich, Germany.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927, Cowles Foundation for Research in Economics, Yale University.
    4. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2017. "Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 683-697, July.
    5. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    6. Blaise Melly und Kaspar Wüthrich, 2016. "Local quantile treatment effects," Diskussionsschriften dp1605, Universitaet Bern, Departement Volkswirtschaft.
    7. Davide Viviano & Kaspar Wuthrich & Paul Niehaus, 2021. "(When) should you adjust inferences for multiple hypothesis testing?," Papers 2104.13367, arXiv.org.
    8. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    9. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    10. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927R, Cowles Foundation for Research in Economics, Yale University, revised Apr 2014.
    11. Bitler, Marianne P. & Gelbach, Jonah B. & Hoynes, Hilary W., 2008. "Distributional impacts of the Self-Sufficiency Project," Journal of Public Economics, Elsevier, vol. 92(3-4), pages 748-765, April.
    12. Hartley, Robert Paul & Lamarche, Carlos, 2018. "Behavioral responses and welfare reform: Evidence from a randomized experiment," Labour Economics, Elsevier, vol. 54(C), pages 135-151.
    13. Callaway, Brantly, 2021. "Bounds on distributional treatment effect parameters using panel data with an application on job displacement," Journal of Econometrics, Elsevier, vol. 222(2), pages 861-881.
    14. Thoresen, Thor O. & Vattø, Trine E., 2015. "Validation of the discrete choice labor supply model by methods of the new tax responsiveness literature," Labour Economics, Elsevier, vol. 37(C), pages 38-53.
    15. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    16. Callaway, Brantly, 2021. "Bounds on distributional treatment effect parameters using panel data with an application on job displacement," Journal of Econometrics, Elsevier, vol. 222(2), pages 861-881.
    17. Sergio Firpo & Cristine Pinto, 2016. "Identification and Estimation of Distributional Impacts of Interventions Using Changes in Inequality Measures," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(3), pages 457-486, April.
    18. Ferraro, Paul J. & Miranda, Juan José, 2013. "Heterogeneous treatment effects and mechanisms in information-based environmental policies: Evidence from a large-scale field experiment," Resource and Energy Economics, Elsevier, vol. 35(3), pages 356-379.
    19. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    20. Vishal Kamat, 2017. "Identification of Program Access Effects with an Application to Head Start," Papers 1711.02048, arXiv.org, revised Oct 2020.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs
    • J22 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Time Allocation and Labor Supply

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:22950. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.