IDEAS home Printed from https://ideas.repec.org/a/tpr/restat/v99y2017i4p683-697.html
   My bibliography  Save this article

Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment

Author

Listed:
  • Marianne P. Bitler

    (University of California, Davis and NBER)

  • Jonah B. Gelbach

    (University of Pennsylvania Law School)

  • Hilary W. Hoynes

    (University of California, Berkeley and NBER)

Abstract

We assess whether welfare reform affects earnings only through mean impacts that are constant within but vary across subgroups. This is important because researchers interested in treatment effect heterogeneity typically focus on estimating mean impacts that only vary across subgroups. Using a novel approach to simulating treatment group earnings under the constant mean impacts within subgroup model, we find this model does a poor job of capturing treatment effect heterogeneity for Connecticut's Jobs First welfare reform experiment. Notably, ignoring within-group heterogeneitywould lead one to miss evidence that treatment effects are consistent with basic labor supply theory.

Suggested Citation

  • Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2017. "Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 683-697, July.
  • Handle: RePEc:tpr:restat:v:99:y:2017:i:4:p:683-697
    as

    Download full text from publisher

    File URL: http://www.mitpressjournals.org/doi/pdf/10.1162/REST_a_00662
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anderson, Michael L, 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt15n8j26f, Department of Agricultural & Resource Economics, UC Berkeley.
    2. Patrick Kline & Melissa Tartari, 2016. "Bounding the Labor Supply Responses to a Randomized Welfare Experiment: A Revealed Preference Approach," American Economic Review, American Economic Association, vol. 106(4), pages 972-1014, April.
    3. Oliver Linton & Esfandiar Maasoumi & Yoon-Jae Whang, 2005. "Consistent Testing for Stochastic Dominance under General Sampling Schemes," Review of Economic Studies, Oxford University Press, vol. 72(3), pages 735-765.
    4. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    5. Sergio Firpo, 2007. "Efficient Semiparametric Estimation of Quantile Treatment Effects," Econometrica, Econometric Society, vol. 75(1), pages 259-276, January.
    6. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    7. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 487-535.
    8. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    9. Sergio Firpo & Nicole M. Fortin & Thomas Lemieux, 2009. "Unconditional Quantile Regressions," Econometrica, Econometric Society, vol. 77(3), pages 953-973, May.
    10. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    11. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    12. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    13. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    14. Roger Koenker & Zhijie Xiao, 2002. "Inference on the Quantile Regression Process," Econometrica, Econometric Society, vol. 70(4), pages 1583-1612, July.
    15. Abadie A., 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 284-292, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. Sam Watson’s journal round up for 16th October 2017
      by Sam Watson in The Academic Health Economists' Blog on 2017-10-16 16:00:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Callaway, Brantly, 2021. "Bounds on distributional treatment effect parameters using panel data with an application on job displacement," Journal of Econometrics, Elsevier, vol. 222(2), pages 861-881.
    2. Lazuka, Volha, 2017. "The lasting health and income effects of public health formation in Sweden," Lund Papers in Economic History 153, Lund University, Department of Economic History.
    3. Robert Garlick, 2018. "Academic Peer Effects with Different Group Assignment Policies: Residential Tracking versus Random Assignment," American Economic Journal: Applied Economics, American Economic Association, vol. 10(3), pages 345-369, July.
    4. Alejandro Sanchez-Becerra, 2023. "Robust inference for the treatment effect variance in experiments using machine learning," Papers 2306.03363, arXiv.org.
    5. Xiao, Zhijie & Xu, Lan, 2019. "What do mean impacts miss? Distributional effects of corporate diversification," Journal of Econometrics, Elsevier, vol. 213(1), pages 92-120.
    6. Afrouz Azadikhah Jahromi & Brantly Callaway, 2022. "Heterogeneous Effects of Job Displacement on Earnings," Empirical Economics, Springer, vol. 62(1), pages 213-245, January.
    7. Gibbons Charles E. & Suárez Serrato Juan Carlos & Urbancic Michael B., 2019. "Broken or Fixed Effects?," Journal of Econometric Methods, De Gruyter, vol. 8(1), pages 1-12, January.
    8. Maike Hohberg & Peter Pütz & Thomas Kneib, 2020. "Treatment effects beyond the mean using distributional regression: Methods and guidance," PLOS ONE, Public Library of Science, vol. 15(2), pages 1-29, February.
    9. Deirdre Bloome & Daniel Schrage, 2021. "Covariance Regression Models for Studying Treatment Effect Heterogeneity Across One or More Outcomes: Understanding How Treatments Shape Inequality," Sociological Methods & Research, , vol. 50(3), pages 1034-1072, August.
    10. Xavier D’Haultfoeuille & Pauline Givord, 2014. "La régression quantile en pratique," Économie et Statistique, Programme National Persée, vol. 471(1), pages 85-111.
    11. Chung, EunYi & Olivares, Mauricio, 2021. "Permutation test for heterogeneous treatment effects with a nuisance parameter," Journal of Econometrics, Elsevier, vol. 225(2), pages 148-174.
    12. Jeffrey Smith, 2022. "Treatment Effect Heterogeneity," Evaluation Review, , vol. 46(5), pages 652-677, October.
    13. Likai Chen & Georg Keilbar & Liangjun Su & Weining Wang, 2023. "Tests for Many Treatment Effects in Regression Discontinuity Panel Data Models," Papers 2312.01162, arXiv.org.
    14. Brigham R. Frandsen & Lars J. Lefgren, 2021. "Partial identification of the distribution of treatment effects with an application to the Knowledge is Power Program (KIPP)," Quantitative Economics, Econometric Society, vol. 12(1), pages 143-171, January.
    15. Horacio Alvarez Marinelli & Samuel Berlinski & Matias Busso, 2024. "Remedial Education: Evidence from a Sequence of Experiments in Colombia," Journal of Human Resources, University of Wisconsin Press, vol. 59(1), pages 141-174.
    16. Strittmatter, Anthony, 2019. "Heterogeneous earnings effects of the job corps by gender: A translated quantile approach," Labour Economics, Elsevier, vol. 61(C).
    17. Matthew L. Comey & Amanda R. Eng & Zhuan Pei, 2022. "Supercompliers," Papers 2212.14105, arXiv.org, revised Aug 2023.
    18. Jaime Ramirez-Cuellar, 2023. "Testing for idiosyncratic Treatment Effect Heterogeneity," Papers 2304.01141, arXiv.org.
    19. Strittmatter, Anthony, 2023. "What is the value added by using causal machine learning methods in a welfare experiment evaluation?," Labour Economics, Elsevier, vol. 84(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Victor Chernozhukov & Iván Fernández‐Val & Blaise Melly, 2013. "Inference on Counterfactual Distributions," Econometrica, Econometric Society, vol. 81(6), pages 2205-2268, November.
    3. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    4. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2022. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1552-1568, October.
    5. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    6. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    7. Sokbae Lee & Yoon-Jae Whang, 2009. "Nonparametric Tests of Conditional Treatment Effects," Cowles Foundation Discussion Papers 1740, Cowles Foundation for Research in Economics, Yale University.
    8. Qu, Zhongjun & Yoon, Jungmo, 2015. "Nonparametric estimation and inference on conditional quantile processes," Journal of Econometrics, Elsevier, vol. 185(1), pages 1-19.
    9. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).
    10. Ferreira, Francisco H. G. & Firpo, Sergio & Galvao, Antonio F., 2017. "Estimation and Inference for Actual and Counterfactual Growth Incidence Curves," IZA Discussion Papers 10473, Institute of Labor Economics (IZA).
    11. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    12. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2021. "A Nonparametric Test for Testing Heterogeneity in Conditional Quantile Treatment Effects," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202117, University of Kansas, Department of Economics, revised Aug 2021.
    13. Ozkan Eren & Serkan Ozbeklik, 2014. "Who Benefits From Job Corps? A Distributional Analysis Of An Active Labor Market Program," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 586-611, June.
    14. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    15. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2020. "Inferences for Partially Conditional Quantile Treatment Effect Model," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202005, University of Kansas, Department of Economics, revised Feb 2020.
    16. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    17. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    18. Callaway, Brantly & Li, Tong & Oka, Tatsushi, 2018. "Quantile treatment effects in difference in differences models under dependence restrictions and with only two time periods," Journal of Econometrics, Elsevier, vol. 206(2), pages 395-413.
    19. Frandsen, Brigham R. & Frölich, Markus & Melly, Blaise, 2012. "Quantile treatment effects in the regression discontinuity design," Journal of Econometrics, Elsevier, vol. 168(2), pages 382-395.
    20. Sakos, Grayson & Cerulli, Giovanni & Garbero, Alessandra, 2021. "Beyond the ATE: Idiosyncratic Effect Estimation to Uncover Distributional Impacts Results from 17 Impact Evaluations," 2021 Annual Meeting, August 1-3, Austin, Texas 314017, Agricultural and Applied Economics Association.

    More about this item

    JEL classification:

    • H75 - Public Economics - - State and Local Government; Intergovernmental Relations - - - State and Local Government: Health, Education, and Welfare
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs
    • J18 - Labor and Demographic Economics - - Demographic Economics - - - Public Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tpr:restat:v:99:y:2017:i:4:p:683-697. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kelly McDougall (email available below). General contact details of provider: https://direct.mit.edu/journals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.