IDEAS home Printed from https://ideas.repec.org/p/ifs/cemmap/45-15.html
   My bibliography  Save this paper

Inference under covariate-adaptive randomization

Author

Listed:
  • Federico A. Bugni

    (Institute for Fiscal Studies and Duke University)

  • Ivan A. Canay

    (Institute for Fiscal Studies and Northwestern University)

  • Azeem M. Shaikh

    (Institute for Fiscal Studies and University of Chicago)

Abstract

This paper studies inference for the average treatment effect in randomized controlled trials with covariate-adaptive randomization. Here, by covariate-adaptive randomization, we mean randomization schemes that ?rst stratify according to baseline covariates and then assign treatment status so as to achieve 'balance' within each stratum. Such schemes include, for example, Efron's biased-coin design and strati?ed block randomization. When testing the null hypothesis that the average treatment effect equals a pre-speci?ed value in such settings, we ?rst show that the usual two-sample t-test is conservative in the sense that it has limiting rejection probability under the null hypothesis no greater than and typically strictly less than the nominal level. In a simulation study, we ?nd that the rejection probability may in fact be dramatically less than the nominal level. We show further that these same conclusions remain true for a naïve permutation test, but that a modi?ed version of the permutation test yields a test that is non-conservative in the sense that its limiting rejection probability under the null hypothesis equals the nominal level. The modi?ed version of the permutation test has the additional advantage that it has rejection probability exactly equal to the nominal level for some distributions satisfying the null hypothesis. Finally, we show that the usual t-test (on the coefficient on treatment assignment) in a linear regression of outcomes on treatment assignment and indicators for each of the strata yields a non-conservative test as well. In a simulation study, we ?nd that the non-conservative tests have substantially greater power than the usual two-sample t-test.

Suggested Citation

  • Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2015. "Inference under covariate-adaptive randomization," CeMMAP working papers CWP45/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
  • Handle: RePEc:ifs:cemmap:45/15
    as

    Download full text from publisher

    File URL: https://www.ifs.org.uk/uploads/cemmap/wps/cwp451515.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Guido W. Imbens & Michal Kolesár, 2016. "Robust Standard Errors in Small Samples: Some Practical Advice," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 701-712, October.
    2. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2015. "Inference under covariate-adaptive randomization," CeMMAP working papers CWP45/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    3. Berry, James & Karlan, Dean & Pradhan, Menno, 2018. "The Impact of Financial Education for Youth in Ghana," World Development, Elsevier, vol. 102(C), pages 71-89.
    4. Michael Callen & Saad Gulzar & Ali Hasanain & Yasir Khan & Arman Rezaee, 2015. "Personalities and Public Sector Performance: Evidence from a Health Experiment in Pakistan," NBER Working Papers 21180, National Bureau of Economic Research, Inc.
    5. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    6. Rodrigo Pinto & Azeem Shaikh & Adam Yavitz & James Heckman, 2010. "Inference with Imperfect Randomization: The Case of the Perry Preschool Program," 2010 Meeting Papers 1336, Society for Economic Dynamics.
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, Elsevier.
    8. Esther Duflo & Pascaline Dupas & Michael Kremer, 2015. "Education, HIV, and Early Fertility: Experimental Evidence from Kenya," American Economic Review, American Economic Association, vol. 105(9), pages 2757-2797, September.
    9. Alberto Chong & Isabelle Cohen & Erica Field & Eduardo Nakasone & Maximo Torero, 2016. "Iron Deficiency and Schooling Attainment in Peru," American Economic Journal: Applied Economics, American Economic Association, vol. 8(4), pages 222-255, October.
    10. Jun Shao & Xinxin Yu & Bob Zhong, 2010. "A theory for testing hypotheses under covariate-adaptive randomization," Biometrika, Biometrika Trust, vol. 97(2), pages 347-360.
    11. Rosenbaum, Paul R., 2007. "Interference Between Units in Randomized Experiments," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 191-200, March.
    12. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    13. Ali Hasanain & Saad Gulzar & Arman Rezaee & Yasir Khan, 2015. "Personalities and Public Sector Performance: Evidence from a Health Experiment in Pakistan," Working Papers id:6690, eSocialSciences.
    14. Janssen, Arnold, 1997. "Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem," Statistics & Probability Letters, Elsevier, vol. 36(1), pages 9-21, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2015. "Inference under covariate-adaptive randomization," CeMMAP working papers CWP45/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    2. repec:tsj:stataj:y:17:y:2017:i:3:p:630-651 is not listed on IDEAS
    3. Gonzalo Vazquez-Bare, 2017. "Identification and Estimation of Spillover Effects in Randomized Experiments," Papers 1711.02745, arXiv.org, revised Jun 2019.
    4. Isaiah Andrews & Emily Oster, 2017. "Weighting for External Validity," NBER Working Papers 23826, National Bureau of Economic Research, Inc.
    5. John List & Azeem Shaikh & Yang Xu, 2016. "Multiple Hypothesis Testing in Experimental Economics," Artefactual Field Experiments 00402, The Field Experiments Website.
    6. Ivan A. Canay & Vishal Kamat, 2015. "Approximate permutation tests and induced order statistics in the regression discontinuity design," CeMMAP working papers CWP27/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    7. Abhijit Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2017. "A Theory of Experimenters," NBER Working Papers 23867, National Bureau of Economic Research, Inc.
    8. Tymon S{l}oczy'nski, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," Papers 1810.01576, arXiv.org, revised Dec 2018.
    9. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    10. repec:eee:deveng:v:1:y:2016:i:c:p:12-25 is not listed on IDEAS
    11. Vishal Kamat, 2017. "Identification with Latent Choice Sets," Papers 1711.02048, arXiv.org, revised Aug 2019.
    12. Abhijit Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2017. "A Theory of Experimenters," CESifo Working Paper Series 6678, CESifo Group Munich.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ifs:cemmap:45/15. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Emma Hyman). General contact details of provider: http://edirc.repec.org/data/cmifsuk.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.