IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2607.16982.html

When and How to Pilot: Design Rules for Two-Wave Experiments

Author

Listed:
  • Juan C. Yamin

Abstract

Experimenters often run pilots, but how much a small pilot should shape the main-wave design has no settled answer. This paper shows how noisy pilot evidence should guide treatment assignment probabilities in two-wave experiments. Two canonical rules mark the extremes. Balanced assignment guards against worst cases but ignores evidence that one arm is noisier. Feasible Neyman allocation adapts, but with a finite pilot it can overreact to noise, producing arbitrarily large precision losses. I propose a Conditional Minimax Regret (CMR) rule that minimizes worst-case regret over a finite-sample confidence set for the treatment and control variances. CMR retains balance's worst-case protection with high probability, converges to the Neyman allocation as the pilot grows, and attains the minimax-regret rate up to constants. It extends to multi-arm and stratified designs, and simulations calibrated to four field experiments show it avoids feasible Neyman's severe small-pilot losses while matching its large-pilot gains.

Suggested Citation

  • Juan C. Yamin, 2026. "When and How to Pilot: Design Rules for Two-Wave Experiments," Papers 2607.16982, arXiv.org, revised Aug 2026.
    • Handle: RePEc:arx:papers:2607.16982
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2607.16982
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Peter Hull, 2026. "Optimal Experimental Design and Estimation when Potential Outcomes are Bounded," Papers 2608.09812, arXiv.org, revised Aug 2026.
    2. Jinyong Hahn & Keisuke Hirano & Dean Karlan, 2011. "Adaptive Experimental Design Using the Propensity Score," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(1), pages 96-108, January.
    3. Stoye, Jörg, 2009. "Minimax regret treatment choice with finite samples," Journal of Econometrics, Elsevier, vol. 151(1), pages 70-81, July.
    4. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    5. Max Tabord-Meehan, 2023. "Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2646-2673.
    6. Martin Abel & Rulof Burger & Patrizio Piraino, 2020. "The Value of Reference Letters: Experimental Evidence from South Africa," American Economic Journal: Applied Economics, American Economic Association, vol. 12(3), pages 40-71, July.
    7. Kentaro Kawato & Shosei Sakaguchi, 2026. "Prior-Free Sample Size Design for Test-and-Roll Experiments," Papers 2605.02414, arXiv.org.
    8. Max Cytrynbaum, 2021. "Fine Stratification of Survey Experiments," Papers 2111.08157, arXiv.org, revised Sep 2026.
    9. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    10. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    11. Cheaheon Lim & Yechan Park, 2026. "Dynamically Consistent Statistical Decisions," Papers 2607.10519, arXiv.org.
    12. Cai, Yong & Rafi, Ahnaf, 2024. "On the performance of the Neyman Allocation with small pilots," Journal of Econometrics, Elsevier, vol. 242(1).
    13. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    14. Victor Chernozhukov & Sokbae Lee & Adam M. Rosen & Liyang Sun, 2025. "Policy Learning with Confidence," Papers 2502.10653, arXiv.org, revised Jan 2026.
    15. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    16. Rebecca L. Thornton, 2008. "The Demand for, and Impact of, Learning HIV Status," American Economic Review, American Economic Association, vol. 98(5), pages 1829-1863, December.
    17. Porzio, Giovanni C. & Ragozini, Giancarlo, 2009. "On the stochastic ordering of folded binomials," Statistics & Probability Letters, Elsevier, vol. 79(9), pages 1299-1304, May.
    18. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    19. Raj Chetty & Nathaniel Hendren & Lawrence F. Katz, 2016. "The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment," American Economic Review, American Economic Association, vol. 106(4), pages 855-902, April.
    20. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," American Economic Review, American Economic Association, vol. 112(12), pages 3911-3940, December.
    21. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    2. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    3. Max Cytrynbaum, 2024. "Covariate adjustment in stratified experiments," Quantitative Economics, Econometric Society, vol. 15(4), pages 971-998, November.
    4. Max Cytrynbaum & Fredrik Savje, 2026. "Coupling Designs for Randomized Experiments with Complex Treatments," Papers 2604.09858, arXiv.org, revised Jul 2026.
    5. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    6. Masahiro Kato & Masaaki Imaizumi & Takuya Ishihara & Toru Kitagawa, 2023. "Asymptotically Optimal Fixed-Budget Best Arm Identification with Variance-Dependent Bounds," Papers 2302.02988, arXiv.org, revised Jul 2023.
    7. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    8. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    9. Cai, Yong & Rafi, Ahnaf, 2024. "On the performance of the Neyman Allocation with small pilots," Journal of Econometrics, Elsevier, vol. 242(1).
    10. Jinglong Zhao, 2026. "Optimal Control Variates for Survey Sampling and Causal Inference," Papers 2608.15333, arXiv.org, revised Aug 2026.
    11. Jiang, Liang & Li, Liyao & Miao, Ke & Zhang, Yichong, 2025. "Adjustments with many regressors under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 249(PB).
    12. Bokelmann, Björn & Lessmann, Stefan, 2025. "Heteroscedasticity-aware stratified sampling to improve uplift modeling," European Journal of Operational Research, Elsevier, vol. 325(1), pages 118-131.
    13. Max Cytrynbaum, 2026. "The Limits of Experimental Design: Covariate Balance Beyond Low Dimension," Papers 2608.18057, arXiv.org.
    14. Zikai Li, 2025. "Bridging Stratification and Regression Adjustment: Batch-Adaptive Stratification with Post-Design Adjustment in Randomized Experiments," Papers 2510.22908, arXiv.org.
    15. Jinglong Zhao, 2023. "Adaptive Neyman Allocation," Papers 2309.08808, arXiv.org, revised Jan 2026.
    16. Hirano, Keisuke & Porter, Jack R., 2020. "Asymptotic analysis of statistical decision rules in econometrics," Handbook of Econometrics, in: Steven N. Durlauf & Lars Peter Hansen & James J. Heckman & Rosa L. Matzkin (ed.), Handbook of Econometrics, edition 1, volume 7, chapter 0, pages 283-354, Elsevier.
    17. Masahiro Kato, 2023. "Worst-Case Optimal Multi-Armed Gaussian Best Arm Identification with a Fixed Budget," Papers 2310.19788, arXiv.org, revised Mar 2024.
    18. Sokbae (Simon) Lee & Yoon-Jae Whang, 2009. "Nonparametric tests of conditional treatment effects," CeMMAP working papers CWP36/09, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    19. Laurent Davezies & Guillaume Hollard & Pedro Vergara Merino, 2024. "Revisiting Randomization with the Cube Method," Papers 2407.13613, arXiv.org, revised Jul 2025.
    20. Zhengyuan Zhou & Susan Athey & Stefan Wager, 2023. "Offline Multi-Action Policy Learning: Generalization and Optimization," Operations Research, INFORMS, vol. 71(1), pages 148-183, January.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2607.16982. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.