IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v43y2018i5p568-593.html
   My bibliography  Save this article

Design-Based Estimators for Average Treatment Effects for Multi-Armed RCTs

Author

Listed:
  • Peter Z. Schochet

    (Mathematica Policy Research, Inc.)

Abstract

Design-based methods have recently been developed as a way to analyze randomized controlled trial (RCT) data for designs with a single treatment and control group. This article builds on this framework to develop design-based estimators for evaluations with multiple research groups. Results are provided for a wide range of designs used in education research, including clustered and blocked designs. Because analysis in the multi-armed setting involves pairwise contrasts across the research groups, the key methodological question addressed is: How do the estimators for the two-group design need to be adjusted for multi-armed trials? The critical insight is that in multi-armed trials where the goal is to identify the most effective treatments, the samples for each pairwise contrast are representative of the full set of randomized units, not just of themselves. The implication is that variance terms need to be adjusted slightly under the finite-population framework that can reduce precision, and blocks need to be weighted to reflect the full randomized sample in the block or biases can result. An empirical example using data from a multi-armed education RCT demonstrates the issues.

Suggested Citation

  • Peter Z. Schochet, 2018. "Design-Based Estimators for Average Treatment Effects for Multi-Armed RCTs," Journal of Educational and Behavioral Statistics, , vol. 43(5), pages 568-593, October.
    • Handle: RePEc:sae:jedbes:v:43:y:2018:i:5:p:568-593
    DOI: 10.3102/1076998618786968
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998618786968
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998618786968?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Tirthankar Dasgupta & Natesh S. Pillai & Donald B. Rubin, 2015. "Causal inference from 2-super-K factorial designs by using potential outcomes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(4), pages 727-753, September.
    2. Castleman, Benjamin L. & Page, Lindsay C., 2015. "Summer nudging: Can personalized text messages and peer mentor outreach increase college going among low-income high school graduates?," Journal of Economic Behavior & Organization, Elsevier, vol. 115(C), pages 144-160.
    3. Yang L. & Tsiatis A. A., 2001. "Efficiency Study of Estimators for a Treatment Effect in a Pretest-Posttest Trial," The American Statistician, American Statistical Association, vol. 55, pages 314-321, November.
    4. repec:mpr:mprres:7723 is not listed on IDEAS
    5. Xinran Li & Peng Ding, 2017. "General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1759-1769, October.
    6. Roberto Agodini & Barbara Harris & Sally Atkins-Burnett & Sheila Heaviside & Timothy Novak & Robert Murphy, "undated". "Achievement Effects of Four Early Elementary School Math Curricula: Findings from First Graders in 39 Schools," Mathematica Policy Research Reports 467194cd2d6f4cbaba7e23745, Mathematica Policy Research.
    7. Roberto Agodini, 2009. "Achievement Effects of Four Early Elementary School Math Curricula: Findings from First Graders in 39 Schools (Conference Paper)," Mathematica Policy Research Reports a08d661306a843fa89aff4c30, Mathematica Policy Research.
    8. repec:mpr:mprres:6192 is not listed on IDEAS
    9. repec:mpr:mprres:6574 is not listed on IDEAS
    10. Hansen, Christian B., 2007. "Asymptotic properties of a robust variance matrix estimator for panel data when T is large," Journal of Econometrics, Elsevier, vol. 141(2), pages 597-620, December.
    11. repec:mpr:mprres:6573 is not listed on IDEAS
    12. Peter Z. Schochet, "undated". "Is Regression Adjustment Supported by the Neyman Model for Causal Inference? (Presentation)," Mathematica Policy Research Reports abfc39d59c714499b2fe42f68, Mathematica Policy Research.
    13. Peter Z. Schochet, "undated". "Is Regression Adjustment Supported By the Neyman Model for Causal Inference?," Mathematica Policy Research Reports 782da2242fba458eb61752f96, Mathematica Policy Research.
    14. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    15. repec:mpr:mprres:7337 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Peter Z. Schochet, 2020. "Analyzing Grouped Administrative Data for RCTs Using Design-Based Methods," Journal of Educational and Behavioral Statistics, , vol. 45(1), pages 32-57, February.
    2. Lu, Jiannan, 2016. "On randomization-based and regression-based inferences for 2K factorial designs," Statistics & Probability Letters, Elsevier, vol. 112(C), pages 72-78.
    3. Peter Z. Schochet, 2013. "Estimators for Clustered Education RCTs Using the Neyman Model for Causal Inference," Journal of Educational and Behavioral Statistics, , vol. 38(3), pages 219-238, June.
    4. Peter Z. Schochet, "undated". "Multi-Armed RCTs: A Design-Based Framework," Mathematica Policy Research Reports eedf2eac4d4c4d8e869052c1d, Mathematica Policy Research.
    5. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    6. repec:mpr:mprres:6094 is not listed on IDEAS
    7. Joel A. Middleton, 2021. "Unifying Design-based Inference: On Bounding and Estimating the Variance of any Linear Estimator in any Experimental Design," Papers 2109.09220, arXiv.org.
    8. Peter Z. Schochet, 2010. "The Late Pretest Problem in Randomized Control Trials of Education Interventions," Journal of Educational and Behavioral Statistics, , vol. 35(4), pages 379-406, August.
    9. Peter Z. Schochet, "undated". "The Late Pretest Problem in Randomized Control Trials of Education Interventions," Mathematica Policy Research Reports fb514df5dbb84a5dbea79865c, Mathematica Policy Research.
    10. Peter Z. Schochet, "undated". "Technical Methods Report: Statistical Power for Regression Discontinuity Designs in Education Evaluations," Mathematica Policy Research Reports 61fb6c057561451a8a6074508, Mathematica Policy Research.
    11. repec:mpr:mprres:6372 is not listed on IDEAS
    12. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    13. Fangzhou Su & Peng Ding, 2021. "Model‐assisted analyses of cluster‐randomized experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 994-1015, November.
    14. Tim Kautz & Peter Z. Schochet & Charles Tilley, "undated". "Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation," Mathematica Policy Research Reports b7656ddce20f4007b71836e99, Mathematica Policy Research.
    15. repec:mpr:mprres:7638 is not listed on IDEAS
    16. Clément de Chaisemartin, 2022. "Trading-off Bias and Variance in Stratified Experiments and in Staggered Adoption Designs, Under a Boundedness Condition on the Magnitude of the Treatment Effect," Working Papers hal-03873919, HAL.
    17. repec:mpr:mprres:6965 is not listed on IDEAS
    18. repec:mpr:mprres:6286 is not listed on IDEAS
    19. Douglas H. Clements & Roberto Agodini & Barbara Harris, "undated". "Instructional Practices and Student Math Achievement: Correlations from a Study of Math Curricula (Issue Brief)," Mathematica Policy Research Reports 1e18ce07a82f42479daa643f7, Mathematica Policy Research.
    20. repec:mpr:mprres:7907 is not listed on IDEAS
    21. Xiaokang Luo & Tirthankar Dasgupta & Minge Xie & Regina Y. Liu, 2021. "Leveraging the Fisher randomization test using confidence distributions: Inference, combination and fusion learning," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(4), pages 777-797, September.
    22. repec:mpr:mprres:7273 is not listed on IDEAS
    23. Kenneth Fortson & Natalya Verbitsky-Savitz & Emma Kopa & Philip Gleason, 2012. "Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates," Mathematica Policy Research Reports 27f871b5b7b94f3a80278a593, Mathematica Policy Research.
    24. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    25. John Deke, 2016. "Design and Analysis Considerations for Cluster Randomized Controlled Trials That Have a Small Number of Clusters," Evaluation Review, , vol. 40(5), pages 444-486, October.
    26. Kimberly Boller & Sally Atkins-Burnett & Elizabeth M. Malone & Gail P. Baxter & Jerry West, "undated". "Compendium of Student, Teacher, and Classroom Measures Used in NCEE Evaluations of Educational Interventions, Volume I: Measures Selection Approaches and Compendium Development Methods," Mathematica Policy Research Reports 07a8a75ba5634b3a813e22bc6, Mathematica Policy Research.
    27. John Deke & Lisa Dragoset, "undated". "Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials," Mathematica Policy Research Reports a4f1d03eb7bf427a8983d4736, Mathematica Policy Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:43:y:2018:i:5:p:568-593. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.