IDEAS home Printed from
   My bibliography  Save this paper

Inference under Covariate-Adaptive Randomization


  • Federico A. Bugni

    (Institute for Fiscal Studies and Duke University)

  • Ivan A. Canay

    (Institute for Fiscal Studies and Northwestern University)

  • Azeem M. Shaikh

    (Institute for Fiscal Studies and University of Chicago)


This paper studies inference for the average treatment eff ect in randomized controlled trials with covariate-adaptive randomization. Here, by covariate-adaptive randomization, we mean randomization schemes that first stratify according to baseline covariates and then assign treatment status so as to achieve "balance" within each stratum. Such schemes include, for example, Efron's biased-coin design and strati ed block randomization. When testing the null hypothesis that the average treatment eff ect equals a pre-speci fied value in such settings, we fi rst show that the usual two-sample t-test is conservative in the sense that it has limiting rejection probability under the null hypothesis no greater than and typically strictly less than the nominal level. In a simulation study, we fi nd that the rejection probability may in fact be dramatically less than the nominal level. We show further that these same conclusions remain true for a naïve permutation test, but that a modi fied version of the permutation test yields a test that is non-conservative in the sense that its limiting rejection probability under the null hypothesis equals the nominal level for a wide variety of randomization schemes. The modi fied version of the permutation test has the additional advantage that it has rejection probability exactly equal to the nominal level for some distributions satisfying the null hypothesis and some randomization schemes. Finally, we show that the usual t-test (on the coefficient on treatment assignment) in a linear regression of outcomes on treatment assignment and indicators for each of the strata yields a non-conservative test as well under even weaker assumptions on the randomization scheme. In a simulation study, we fi nd that the non-conservative tests have substantially greater power than the usual two-sample t-test.

Suggested Citation

  • Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2016. "Inference under Covariate-Adaptive Randomization," CeMMAP working papers CWP21/16, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  • Handle: RePEc:ifs:cemmap:21/16

    Download full text from publisher

    File URL:
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. Guido W. Imbens & Michal Kolesár, 2016. "Robust Standard Errors in Small Samples: Some Practical Advice," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 701-712, October.
    2. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    3. Berry, James & Karlan, Dean & Pradhan, Menno, 2018. "The Impact of Financial Education for Youth in Ghana," World Development, Elsevier, vol. 102(C), pages 71-89.
    4. Ali Hasanain & Saad Gulzar & Arman Rezaee & Yasir Khan, 2015. "Personalities and Public Sector Performance: Evidence from a Health Experiment in Pakistan," Working Papers id:6690, eSocialSciences.
    5. Rodrigo Pinto & Azeem Shaikh & Adam Yavitz & James Heckman, 2010. "Inference with Imperfect Randomization: The Case of the Perry Preschool Program," 2010 Meeting Papers 1336, Society for Economic Dynamics.
    6. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    8. Esther Duflo & Pascaline Dupas & Michael Kremer, 2015. "Education, HIV, and Early Fertility: Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 105(9), pages 2757-2797, September.
    9. Alberto Chong & Isabelle Cohen & Erica Field & Eduardo Nakasone & Maximo Torero, 2016. "Iron Deficiency and Schooling Attainment in Peru," American Economic Journal: Applied Economics, American Economic Association, vol. 8(4), pages 222-255, October.
    10. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    11. Jun Shao & Xinxin Yu & Bob Zhong, 2010. "A theory for testing hypotheses under covariate-adaptive randomization," Biometrika, Biometrika Trust, vol. 97(2), pages 347-360.
    12. Rosenbaum, Paul R., 2007. "Interference Between Units in Randomized Experiments," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 191-200, March.
    13. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, November.
    14. Janssen, Arnold, 1997. "Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem," Statistics & Probability Letters, Elsevier, vol. 36(1), pages 9-21, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2019. "Inference under covariate‐adaptive randomization with multiple treatments," Quantitative Economics, Econometric Society, vol. 10(4), pages 1747-1785, November.
    2. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    3. Yuehao Bai & Joseph P. Romano & Azeem M. Shaikh, 2022. "Inference in Experiments With Matched Pairs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(540), pages 1726-1737, October.
    4. Yichong Zhang & Xin Zheng, 2020. "Quantile treatment effects and bootstrap inference under covariate‐adaptive randomization," Quantitative Economics, Econometric Society, vol. 11(3), pages 957-982, July.
    5. Max Tabord-Meehan, 2018. "Stratification Trees for Adaptive Randomization in Randomized Controlled Trials," Papers 1806.05127,, revised Jul 2022.
    6. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845,
    7. Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
    8. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    9. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    10. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials [Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco]," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    11. Patrizia Lattarulo & Marco Mariani & Laura Razzolini, 2017. "Nudging museums attendance: a field experiment with high school teens," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 41(3), pages 259-277, August.
    12. Tong Wang & Wei Ma, 2021. "The impact of misclassification on covariate‐adaptive randomized clinical trials," Biometrics, The International Biometric Society, vol. 77(2), pages 451-464, June.
    13. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    14. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    15. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    16. Chung, EunYi & Olivares, Mauricio, 2021. "Permutation test for heterogeneous treatment effects with a nuisance parameter," Journal of Econometrics, Elsevier, vol. 225(2), pages 148-174.
    17. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    18. Rachel Cassidy & Anaya Dam & Wendy Janssens & Umair Kiani & Karlijn Morsink, 2022. "Father of the Bride, or Steel Magnolias? Targeting men, women or both to reduce child marriage," Tinbergen Institute Discussion Papers 22-087/V, Tinbergen Institute.
    19. Bugni, Federico A. & Gao, Mengsi, 2023. "Inference under covariate-adaptive randomization with imperfect compliance," Journal of Econometrics, Elsevier, vol. 237(1).
    20. Asresu Yitayew & Awudu Abdulai & Yigezu A Yigezu, 2022. "Improved agricultural input delivery systems for enhancing technology adoption: evidence from a field experiment in Ethiopia," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 49(3), pages 527-556.

    More about this item


    Covariate-adaptive randomization; strati ed block randomization; Efron's biased-coin design; treatment assignment; randomized controlled trial; permutation test; two-sample t-test; strata xed e ects;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ifs:cemmap:21/16. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Emma Hyman (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.