IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/138897.html

Robust sequential experimental design for A/B testing

Author

Listed:
  • Wen, Qianglin
  • Wu, Xiangkun
  • Shi, Chengchun
  • Li, Ting
  • Tang, Niansheng
  • Zhang, Yingying
  • Zhu, Hongtu

Abstract

Experimental design has emerged as a powerful approach for improving the sample efficiency of A/B testing, yet existing designs rely critically on correctly specified models. We study robust sequential experimental design under model mis specification and develop a unified framework that covers both contextual bandit and dynamic settings. Theoretically, we prove that our de sign bounds the worst-case mean squared error of the estimated treatment effect. Empirically, we demonstrate the effectiveness of the proposed approach using synthetic and real-world datasets from a leading technology company.

Suggested Citation

  • Wen, Qianglin & Wu, Xiangkun & Shi, Chengchun & Li, Ting & Tang, Niansheng & Zhang, Yingying & Zhu, Hongtu, 2026. "Robust sequential experimental design for A/B testing," LSE Research Online Documents on Economics 138897, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:138897
    as

    Download full text from publisher

    File URL: https://researchonline.lse.ac.uk/id/eprint/138897/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Guillaume W Basse & Yi Ding & Panos Toulis, 2023. "Minimax designs for causal effects in temporal experiments with treatment habituation," Biometrika, Biometrika Trust, vol. 110(1), pages 155-168.
    2. Douglas P. Wiens, 2009. "Robust discrimination designs," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(4), pages 805-829, September.
    3. Cai, Yong & Rafi, Ahnaf, 2024. "On the performance of the Neyman Allocation with small pilots," Journal of Econometrics, Elsevier, vol. 242(1).
    4. Luo, Shikai & Yang, Ying & Shi, Chengchun & Yao, Fang & Ye, Jieping & Zhu, Hongtu, 2024. "Policy evaluation for temporal and/or spatial dependent experiments," LSE Research Online Documents on Economics 122741, London School of Economics and Political Science, LSE Library.
    5. Linglong Kong & Douglas P. Wiens, 2015. "Model-Robust Designs for Quantile Regression," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(509), pages 233-245, March.
    6. Shi, Chengchun & Luo, Shikai & Zhu, Hongtu & Song, Rui, 2021. "An online sequential test for qualitative treatment effects," LSE Research Online Documents on Economics 112521, London School of Economics and Political Science, LSE Library.
    7. Ruoxuan Xiong & Alex Chin & Sean J. Taylor, 2024. "Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs," Papers 2406.06768, arXiv.org.
    8. Chengchun Shi & Jin Zhu & Shen Ye & Shikai Luo & Hongtu Zhu & Rui Song, 2024. "Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 273-284, January.
    9. Athey, Susan & Bickel, Peter J. & Chen, Aiyou & Imbens, Guido W. & Pollmann, Michael, 2021. "Semiparametric Estimation of Treatment Effects in Randomized Experiments," Research Papers 3986, Stanford University, Graduate School of Business.
    10. Masahiro Kato & Akihiro Oga & Wataru Komatsubara & Ryo Inokuchi, 2024. "Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices," Papers 2403.03589, arXiv.org, revised Jun 2024.
    11. A. Baldi Antognini & M. Zagoraiou, 2011. "The covariate-adaptive biased coin design for balancing clinical trials in the presence of prognostic factors," Biometrika, Biometrika Trust, vol. 98(3), pages 519-535.
    12. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    13. Shi, Chengchun & Zhu, Jin & Shen, Ye & Luo, Shikai & Zhu, Hongtu & Song, Rui, 2024. "Off-policy confidence interval estimation with confounded Markov decision process," LSE Research Online Documents on Economics 115774, London School of Economics and Political Science, LSE Library.
    14. Adam Kapelner & Abba Krieger, 2014. "Matching on-the-fly: Sequential allocation with higher power and efficiency," Biometrics, The International Biometric Society, vol. 70(2), pages 378-388, June.
    15. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    16. Wei Ma & Ping Li & Li-Xin Zhang & Feifang Hu, 2024. "A New and Unified Family of Covariate Adaptive Randomization Procedures and Their Properties," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 151-162, January.
    17. Viviano, Davide & Bradic, Jelena, 2023. "Synthetic Learner: Model-free inference on treatments over time," Journal of Econometrics, Elsevier, vol. 234(2), pages 691-713.
    18. Jianhua Hu & Hongjian Zhu & Feifang Hu, 2015. "A Unified Family of Covariate-Adjusted Response-Adaptive Designs Based on Efficiency and Ethics," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(509), pages 357-367, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    2. Ke Sun & Linglong Kong & Hongtu Zhu & Chengchun Shi, 2024. "ARMA-Design: Optimal Treatment Allocation Strategies for A/B Testing in Partially Observable Time Series Experiments," Papers 2408.05342, arXiv.org, revised Jan 2025.
    3. Ruoxuan Xiong & Susan Athey & Mohsen Bayati & Guido Imbens, 2024. "Optimal Experimental Design for Staggered Rollouts," Management Science, INFORMS, vol. 70(8), pages 5317-5336, August.
    4. Chonghuan Wang, 2026. "Experimental Design for Matching," Papers 2601.21036, arXiv.org.
    5. Jinglong Zhao, 2023. "Adaptive Neyman Allocation," Papers 2309.08808, arXiv.org, revised Jan 2026.
    6. Zikun Ye & Zhiqi Zhang & Dennis J. Zhang & Heng Zhang & Renyu Zhang, 2026. "Deep Learning-Based Causal Inference for Large-Scale Combinatorial Experiments: Theory and Empirical Evidence," Management Science, INFORMS, vol. 72(7), pages 5611-5634, July.
    7. Adam Kapelner & Abba Krieger, 2023. "A matching procedure for sequential experiments that iteratively learns which covariates improve power," Biometrics, The International Biometric Society, vol. 79(1), pages 216-229, March.
    8. Zhexiao Lin & Peng Ding, 2025. "Unifying regression-based and design-based causal inference in time-series experiments," Papers 2510.22864, arXiv.org.
    9. Shuze Chen & David Simchi-Levi & Chonghuan Wang, 2024. "Improving the Estimation of Lifetime Effects in A/B Testing via Treatment Locality," Papers 2407.19618, arXiv.org, revised Sep 2025.
    10. Michael Thomas & Marcel Goic & Kirthi Kalyanam, 2025. "Long Lags and Large Returns: Experimental Evidence from Advertising to Businesses," Management Science, INFORMS, vol. 71(10), pages 8750-8766, October.
    11. Haoyu Wei, 2025. "Semiparametric Off-Policy Inference for Optimal Policy Values under Possible Non-Uniqueness," Papers 2505.13809, arXiv.org, revised Jun 2026.
    12. Jizhou Liu & Liang Zhong, 2026. "Randomization Tests in Switchback Experiments," Papers 2602.23257, arXiv.org.
    13. Nikhil Bhat & Vivek F. Farias & Ciamac C. Moallemi & Deeksha Sinha, 2020. "Near-Optimal A-B Testing," Management Science, INFORMS, vol. 66(10), pages 4477-4495, October.
    14. Sadegh Shirani & Mohsen Bayati, 2025. "On Evolution-Based Models for Experimentation Under Interference," Papers 2511.21675, arXiv.org.
    15. Li, Zhonghua & Luo, Lan & Wang, Jingshen & Feng, Long, 2025. "Efficient quantile covariate adjusted response adaptive experiments," Journal of Econometrics, Elsevier, vol. 249(PA).
    16. Jacob Carlson & Neil Shephard, 2026. "When are time series predictions causal? The potential system and dynamic causal effects," Papers 2603.20394, arXiv.org.
    17. Hongyu Chen & David Simchi-Levi, 2026. "Efficient Switchback Experiments with Surrogate Variables: Estimation and Experimental Design," Management Science, INFORMS, vol. 72(6), pages 4854-4870, June.
    18. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Brocco & Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org, revised Dec 2025.
    19. Andres Alban & Stephen E. Chick & Spyros I. Zoumpoulis, 2026. "Learning Personalized Treatment Strategies with Predictive and Prognostic Covariates in Adaptive Clinical Trials," Management Science, INFORMS, vol. 72(7), pages 5943-5962, July.
    20. Ke Zhu & Hanzhong Liu, 2023. "Pair‐switching rerandomization," Biometrics, The International Biometric Society, vol. 79(3), pages 2127-2142, September.

    More about this item

    JEL classification:

    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:138897. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.