IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2512.10156.html

Inference for Batched Adaptive Experiments

Author

Listed:
  • Jan Kemper
  • Davud Rostam-Afschar

Abstract

The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, as well as among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.

Suggested Citation

  • Jan Kemper & Davud Rostam-Afschar, 2025. "Inference for Batched Adaptive Experiments," Papers 2512.10156, arXiv.org.
    • Handle: RePEc:arx:papers:2512.10156
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2512.10156
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Max Tabord-Meehan, 2023. "Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2646-2673.
    2. Hadar Avivi & Patrick Kline & Evan Rose & Christopher Walters, 2021. "Adaptive Correspondence Experiments," AEA Papers and Proceedings, American Economic Association, vol. 111, pages 43-48, May.
    3. Maximilian Kasy & Anja Sautmann, 2021. "Adaptive Treatment Assignment in Experiments for Policy Choice," Econometrica, Econometric Society, vol. 89(1), pages 113-132, January.
    4. Gaul, Johannes J. & Keusch, Florian & Rostam-Afschar, Davud & Simon, Thomas, 2024. "Invitation Messages for Business Surveys: A Multi-Armed Bandit Experiment," GLO Discussion Paper Series 1540, Global Labor Organization (GLO).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kemper, Jan & Rostam-Afschar, Davud, 2026. "Earning While Learning: How to Run Batched Bandit Experiments," GLO Discussion Paper Series 1717, Global Labor Organization (GLO).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kemper, Jan & Rostam-Afschar, Davud, 2026. "Earning While Learning: How to Run Batched Bandit Experiments," GLO Discussion Paper Series 1717, Global Labor Organization (GLO).
    2. Masahiro Kato & Masaaki Imaizumi & Takuya Ishihara & Toru Kitagawa, 2023. "Asymptotically Optimal Fixed-Budget Best Arm Identification with Variance-Dependent Bounds," Papers 2302.02988, arXiv.org, revised Jul 2023.
    3. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    4. Harrison H. Li & Art B. Owen, 2023. "Double machine learning and design in batch adaptive experiments," Papers 2309.15297, arXiv.org.
    5. Brian Quistorff & Gentry Johnson, 2020. "Machine Learning for Experimental Design: Methods for Improved Blocking," Papers 2010.15966, arXiv.org.
    6. Chao Qin & Daniel Russo, 2024. "Optimizing Adaptive Experiments: A Unified Approach to Regret Minimization and Best-Arm Identification," Papers 2402.10592, arXiv.org, revised Jul 2024.
    7. Michael Lechner, 2023. "Causal Machine Learning and its use for public policy," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 159(1), pages 1-15, December.
    8. A Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Osman Shami & Alexander Teytelboym, 2024. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," Journal of the European Economic Association, European Economic Association, vol. 22(2), pages 781-836.
    9. Aristotelis Epanomeritakis & Davide Viviano, 2025. "Learning What to Learn: Experimental Design when Combining Experimental with Observational Evidence," Papers 2510.23434, arXiv.org, revised Dec 2025.
    10. Bahety, Girija & Bauhoff, Sebastian & Patel, Dev & Potter, James, 2021. "Texts don’t nudge: An adaptive trial to prevent the spread of COVID-19 in India," Journal of Development Economics, Elsevier, vol. 153(C).
    11. Arthur Charpentier & Romuald Élie & Carl Remlinger, 2023. "Reinforcement Learning in Economics and Finance," Computational Economics, Springer;Society for Computational Economics, vol. 62(1), pages 425-462, June.
    12. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    13. Zhiheng You, 2026. "Using Prior Studies to Design Experiments: An Empirical Bayes Approach," Papers 2602.20581, arXiv.org.
    14. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    15. Martino Banchio & Giacomo Mantegazza, 2022. "Artificial Intelligence and Spontaneous Collusion," Papers 2202.05946, arXiv.org, revised Sep 2023.
    16. Li Harrison H. & Owen Art B., 2024. "Double machine learning and design in batch adaptive experiments," Journal of Causal Inference, De Gruyter, vol. 12(1), pages 1-27.
    17. Molly Offer-Westort & Leah R. Rosenzweig & Susan Athey, 2024. "Battling the coronavirus ‘infodemic’ among social media users in Kenya and Nigeria," Nature Human Behaviour, Nature, vol. 8(5), pages 823-834, May.
    18. Ruicheng Ao & Hongyu Chen & David Simchi-Levi, 2024. "Prediction-Guided Active Experiments," Papers 2411.12036, arXiv.org, revised Nov 2024.
    19. Larissa Fuchs & Matthias Heinz & Pia Pinger & Max Thon, 2024. "How to Attract Talent? Field-Experimental Evidence on Emphasizing Flexibility and Career Opportunities in Job Advertisements," ECONtribute Discussion Papers Series 332, University of Bonn and University of Cologne, Germany.
    20. Xiaoxue Sherry Gao & Glenn W. Harrison & Rusty Tchernis, 2023. "Behavioral welfare economics and risk preferences: a Bayesian approach," Experimental Economics, Springer;Economic Science Association, vol. 26(2), pages 273-303, April.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2512.10156. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.