IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v44y2020i4p262-294.html

Estimating the Distribution of Treatment Effects From Random Design Experiments

Author

Listed:
  • William Rhodes
  • Gerald Gaes

Abstract

Background: Random design experiments are a powerful device for estimating average treatment effects, but evaluators sometimes seek to estimate the distribution of treatment effects. For example, an evaluator might seek to learn the proportion of treated units who benefit from treatment, the proportion who receive no benefit, and the proportion who are harmed by treatment. Method: Imbens and Rubin (I&R) recommend a Bayesian approach to drawing inferences about the distribution of treatment effects. Drawing on the I&R recommendations, this article explains the approach; provides computing algorithms for continuous, binary, ordered and countable outcomes; and offers simulated and real-world illustrations. Results: This article shows how the I&R approach leads to bounded uncertainty intervals for summary measures of the distribution of treatment effects. It clarifies the nature of those bounds and shows that they are typically informative. Conclusions: Despite identification issues, bounded solutions provide useful insight into the distribution of treatment effects. We recommend that evaluators incorporate analyses of the distribution of treatment effects into new studies and that evaluators revisit completed studies to estimate the distribution of treatment effects.

Suggested Citation

  • William Rhodes & Gerald Gaes, 2020. "Estimating the Distribution of Treatment Effects From Random Design Experiments," Evaluation Review, , vol. 44(4), pages 262-294, August.
    • Handle: RePEc:sae:evarev:v:44:y:2020:i:4:p:262-294
    DOI: 10.1177/0193841X20906232
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20906232
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20906232?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    2. Angus Deaton & Nancy Cartwright, 2016. "Understanding and Misunderstanding Randomized Controlled Trials," Working Papers august_25.pdf, Princeton University, Woodrow Wilson School of Public and International Affairs, Research Program in Development Studies..
    3. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    4. Donner, A. & Klar, N., 2004. "Pitfalls of and Controversies in Cluster Randomization Trials," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 416-422.
    5. Constantine E. Frangakis & Donald B. Rubin, 2002. "Principal Stratification in Causal Inference," Biometrics, The International Biometric Society, vol. 58(1), pages 21-29, March.
    6. Robert J. Sampson, 2010. "Gold Standard Myths: Observations on the Experimental Turn in Quantitative Criminology," Journal of Quantitative Criminology, Springer, vol. 26(4), pages 489-500, December.
    7. James J. Heckman & Vytlacil, Edward J., 2007. "Econometric Evaluation of Social Programs, Part I: Causal Models, Structural Models and Econometric Policy Evaluation," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 70, Elsevier.
    8. James Heckman, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    9. Jesse M. Shapiro, 2007. "Do Harsher Prison Conditions Reduce Recidivism? A Discontinuity-based Approach," American Law and Economics Review, American Law and Economics Association, vol. 9(1), pages 1-29.
    10. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Heckman, James & Pinto, Rodrigo, 2024. "Econometric causality: The central role of thought experiments," Journal of Econometrics, Elsevier, vol. 243(1).
    2. Mogstad, Magne & Torgovitsky, Alexander, 2024. "Instrumental variables with unobserved heterogeneity in treatment effects," Handbook of Labor Economics,, Elsevier.
    3. James J. Heckman & Rodrigo Pinto, 2022. "Causality and Econometrics," NBER Working Papers 29787, National Bureau of Economic Research, Inc.
    4. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    5. Myoung-jae Lee, 2017. "Extensive and intensive margin effects in sample selection models: racial effects on wages," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(3), pages 817-839, June.
    6. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(3), pages 247-268, September.
    7. Sokbae Lee & Bernard Salanié, 2018. "Identifying Effects of Multivalued Treatments," Econometrica, Econometric Society, vol. 86(6), pages 1939-1963, November.
    8. Andrea Mercatanti & Fan Li, 2017. "Do debit cards decrease cash demand?: causal inference and sensitivity analysis using principal stratification," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 66(4), pages 759-776, August.
    9. Bonev, Petyo, 2023. "Behavioral Spillovers," Economics Working Paper Series 2303, University of St. Gallen, School of Economics and Political Science.
    10. Huber, Martin & Meier, Jonas & Wallimann, Hannes, 2022. "Business analytics meets artificial intelligence: Assessing the demand effects of discounts on Swiss train tickets," Transportation Research Part B: Methodological, Elsevier, vol. 163(C), pages 22-39.
    11. Rahul Singh & Liyuan Xu & Arthur Gretton, 2020. "Kernel Methods for Causal Functions: Dose, Heterogeneous, and Incremental Response Curves," Papers 2010.04855, arXiv.org, revised Oct 2022.
    12. David H. Dean & Robert C. Dolan & Robert M. Schmidt, 1999. "Evaluating the Vocational Rehabilitation Program Using Longitudinal Data," Evaluation Review, , vol. 23(2), pages 162-189, April.
    13. Simon Calmar Andersen & Louise Beuchert & Phillip Heiler & Helena Skyt Nielsen, 2023. "A Guide to Impact Evaluation under Sample Selection and Missing Data: Teacher's Aides and Adolescent Mental Health," Papers 2308.04963, arXiv.org.
    14. Huber, Martin & Steinmayr, Andreas, 2017. "A Framework for Separating Individual Treatment Effects from Spillover, Interaction, and General Equilibrium Effects," IZA Discussion Papers 10648, IZA Network @ LISER.
    15. Bia, Michela & Flores-Lagunes, Alfonso & Mercatanti, Andrea, 2018. "Evaluation of Language Training Programs in Luxembourg using Principal Stratification," GLO Discussion Paper Series 289, Global Labor Organization (GLO).
    16. Corona, Joshua, 2024. "A Fly-on-the-Wall Study: Measuring Behavior in Social Landscapes," OSF Preprints 4jb2p, Center for Open Science.
    17. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    18. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    19. Deborah A. Cobb‐Clark & Thomas Crossley, 2003. "Econometrics for Evaluations: An Introduction to Recent Developments," The Economic Record, The Economic Society of Australia, vol. 79(247), pages 491-511, December.
    20. Yuta Ota & Takahiro Hoshino & Taisuke Otsu, 2024. "Causal Inference With Auxiliary Observations," Keio-IES Discussion Paper Series 2024-022, Institute for Economics Studies, Keio University.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:44:y:2020:i:4:p:262-294. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.