IDEAS home Printed from https://ideas.repec.org/p/ifs/cemmap/34-17.html
   My bibliography  Save this paper

Inference under covariate-adaptive randomization with multiple treatments

Author

Listed:
  • Federico A. Bugni

    (Institute for Fiscal Studies and Duke University)

  • Ivan A. Canay

    (Institute for Fiscal Studies and Northwestern University)

  • Azeem M. Shaikh

    (Institute for Fiscal Studies and University of Chicago)

Abstract

This paper studies inference in randomized controlled trials with covariate-adaptive randomization when there are multiple treatments. More specifically, we study in this setting inference about the average effect of one or more treatments relative to other treatments or a control. As in Bugni et al. (2017), covariate-adaptive randomization refers to randomization schemes that first stratify according to baseline covariates and then assign treatment status so as to achieve "balance" within each stratum. In contrast to Bugni et al. (2017), however, we allow for the proportion of units being assigned to each of the treatments to vary across strata. We first study the properties of estimators derived from a "fully saturated" linear regression, i.e., a linear regression of the outcome on all interactions between indicators for each of the treatments and indicators for each of the strata. We show that tests based on these estimators using the usual heteroskedasticity-consistent estimator of the asymptotic variance are invalid in the sense that they may have limiting rejection probability under the null hypothesis strictly greater than the nominal level; on the other hand, tests based on these estimators and suitable estimators of the asymptotic variance that we provide are exact in the sense that they have limiting rejection probability under the null hypothesis equal to the nominal level. For the special case in which the target proportion of units being assigned to each of the treatments does not vary across strata, we additionally consider tests based on estimators derived from a linear regression with "strata fixed effects," i.e., a linear regression of the outcome on indicators for each of the treatments and indicators for each of the strata. We show that tests based on these estimators using the usual heteroskedasticity-consistent estimator of the asymptotic variance are conservative in the sense that they have limiting rejection probability under the null hypothesis no greater than and typically strictly less than the nominal level, but tests based on these estimators and suitable estimators of the asymptotic variance that we provide are exact, thereby generalizing results in Bugni et al. (2017) for the case of a single treatment to multiple treatments. A simulation study illustrates the practical relevance of our theoretical results.

Suggested Citation

  • Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2017. "Inference under covariate-adaptive randomization with multiple treatments," CeMMAP working papers CWP34/17, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    • Handle: RePEc:ifs:cemmap:34/17
    as

    Download full text from publisher

    File URL: https://www.ifs.org.uk/uploads/cemmap/wps/CWP341717.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    2. Berry, James & Karlan, Dean & Pradhan, Menno, 2018. "The Impact of Financial Education for Youth in Ghana," World Development, Elsevier, vol. 102(C), pages 71-89.
    3. Ali Hasanain & Saad Gulzar & Arman Rezaee & Yasir Khan, 2015. "Personalities and Public Sector Performance: Evidence from a Health Experiment in Pakistan," Working Papers id:6690, eSocialSciences.
    4. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    5. Rebecca Dizon-Ross, 2018. "Parents' Beliefs About Their Children's Academic Ability: Implications for Educational Investments," NBER Working Papers 24610, National Bureau of Economic Research, Inc.
    6. Esther Duflo & Pascaline Dupas & Michael Kremer, 2015. "Education, HIV, and Early Fertility: Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 105(9), pages 2757-2797, September.
    7. Alberto Chong & Isabelle Cohen & Erica Field & Eduardo Nakasone & Maximo Torero, 2016. "Iron Deficiency and Schooling Attainment in Peru," American Economic Journal: Applied Economics, American Economic Association, vol. 8(4), pages 222-255, October.
    8. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    9. Max Tabord-Meehan, 2018. "Stratification Trees for Adaptive Randomization in Randomized Controlled Trials," Papers 1806.05127, arXiv.org, revised Jul 2022.
    10. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Browne, Oliver R. & Gazze, Ludovica & Greenstone, Michael & Rostapshova, Olga, 2022. "Man vs. Machine : Technological Promise and Political Limits of Automated Regulation Enforcement," CAGE Online Working Paper Series 646, Competitive Advantage in the Global Economy (CAGE).
    2. Christina Korting & Carl Lieberman & Jordan Matsudaira & Zhuan Pei & Yi Shen, 2023. "Visual Inference and Graphical Representation in Regression Discontinuity Designs," The Quarterly Journal of Economics, Oxford University Press, vol. 138(3), pages 1977-2019.
    3. Yujia Gu & Hanzhong Liu & Wei Ma, 2023. "Regression‐based multiple treatment effect estimation under covariate‐adaptive randomization," Biometrics, The International Biometric Society, vol. 79(4), pages 2869-2880, December.
    4. Ruoxuan Xiong & Susan Athey & Mohsen Bayati & Guido Imbens, 2019. "Optimal Experimental Design for Staggered Rollouts," Papers 1911.03764, arXiv.org, revised Sep 2023.
    5. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    6. Heß, Simon & Jaimovich, Dany & Schündeln, Matthias, 2021. "Environmental effects of development programs: Experimental evidence from West African dryland forests," Journal of Development Economics, Elsevier, vol. 153(C).
    7. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    8. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    9. Federico A. Bugni & Ivan A. Canay & Steve McBride, 2023. "Decomposition and Interpretation of Treatment Effects in Settings with Delayed Outcomes," Papers 2302.11505, arXiv.org, revised Oct 2023.
    10. Yuehao Bai & Jizhou Liu & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "On the Efficiency of Finely Stratified Experiments," Papers 2307.15181, arXiv.org, revised Feb 2024.
    11. Guillermo Cruces & Dario Tortarolo & Gonzalo Vazquez-Bare, 2022. "Design of two-stage experiments with an application to spillovers in tax compliance," IFS Working Papers W22/32, Institute for Fiscal Studies.
    12. Maximiliano Lauletta & Martín Rossi & Christian Ruzzier, 2021. "Playing Whac-A-Mole in the Fight against Corruption: Evidence from Random Audits in Brazil," Working Papers 157, Universidad de San Andres, Departamento de Economia, revised Sep 2021.
    13. Liang Jiang & Oliver B. Linton & Haihan Tang & Yichong Zhang, 2022. "Improving Estimation Efficiency via Regression-Adjustment in Covariate-Adaptive Randomizations with Imperfect Compliance," Papers 2201.13004, arXiv.org, revised Jun 2023.
    14. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    15. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    16. Liang Jiang & Liyao Li & Ke Miao & Yichong Zhang, 2023. "Adjustment with Many Regressors Under Covariate-Adaptive Randomizations," Papers 2304.08184, arXiv.org, revised Feb 2024.
    17. Liang Jiang & Xiaobin Liu & Peter C. B. Phillips & Yichong Zhang, 2020. "Bootstrap Inference for Quantile Treatment Effects in Randomized Experiments with Matched Pairs," Papers 2005.11967, arXiv.org, revised May 2021.
    18. Yuehao Bai & Joseph P. Romano & Azeem M. Shaikh, 2022. "Inference in Experiments With Matched Pairs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(540), pages 1726-1737, October.
    19. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    20. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.
    21. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    22. Yuehao Bai & Jizhou Liu & Max Tabord-Meehan, 2022. "Inference for Matched Tuples and Fully Blocked Factorial Designs," Papers 2206.04157, arXiv.org, revised Nov 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    2. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    3. Max Tabord-Meehan, 2018. "Stratification Trees for Adaptive Randomization in Randomized Controlled Trials," Papers 1806.05127, arXiv.org, revised Jul 2022.
    4. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    5. Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
    6. Federico A. Bugni & Mengsi Gao, 2021. "Inference under Covariate-Adaptive Randomization with Imperfect Compliance," Papers 2102.03937, arXiv.org, revised Jul 2023.
    7. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials [Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco]," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    8. Patrizia Lattarulo & Marco Mariani & Laura Razzolini, 2017. "Nudging museums attendance: a field experiment with high school teens," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 41(3), pages 259-277, August.
    9. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    10. Liang Jiang & Xiaobin Liu & Peter C. B. Phillips & Yichong Zhang, 2020. "Bootstrap Inference for Quantile Treatment Effects in Randomized Experiments with Matched Pairs," Papers 2005.11967, arXiv.org, revised May 2021.
    11. Rachel Cassidy & Anaya Dam & Wendy Janssens & Umair Kiani & Karlijn Morsink, 2022. "Father of the Bride, or Steel Magnolias? Targeting men, women or both to reduce child marriage," Tinbergen Institute Discussion Papers 22-087/V, Tinbergen Institute.
    12. Asresu Yitayew & Awudu Abdulai & Yigezu A Yigezu, 2022. "Improved agricultural input delivery systems for enhancing technology adoption: evidence from a field experiment in Ethiopia [The impact of the use of new technologies on farmers’ wheat yield in Et," European Review of Agricultural Economics, Foundation for the European Review of Agricultural Economics, vol. 49(3), pages 527-556.
    13. Yuehao Bai & Jizhou Liu & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "On the Efficiency of Finely Stratified Experiments," Papers 2307.15181, arXiv.org, revised Feb 2024.
    14. Davide Viviano, 2020. "Experimental Design under Network Interference," Papers 2003.08421, arXiv.org, revised Jul 2022.
    15. Yuehao Bai & Meng Hsuan Hsieh & Jizhou Liu & Max Tabord-Meehan, 2022. "Revisiting the Analysis of Matched-Pair and Stratified Experiments in the Presence of Attrition," Papers 2209.11840, arXiv.org, revised Oct 2023.
    16. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    17. Pedro Carneiro & Sokbae (Simon) Lee & Daniel Wilhelm, 2017. "Optimal data collection for randomized control trials," CeMMAP working papers 45/17, Institute for Fiscal Studies.
    18. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    19. Yang, Haoyu & Qin, Yichen & Wang, Fan & Li, Yang & Hu, Feifang, 2023. "Balancing covariates in multi-arm trials via adaptive randomization," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    20. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.

    More about this item

    Keywords

    Covariate-adaptive randomization; multiple treatments; stratifed block randomization; Efron's biased-coin design; treatment assignment; randomized controlled trial; strata fixed effects; saturated regression;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ifs:cemmap:34/17. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Emma Hyman (email available below). General contact details of provider: https://edirc.repec.org/data/cmifsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.