IDEAS home Printed from https://ideas.repec.org/a/eee/jetheo/v210y2023ics002205312300042x.html
   My bibliography  Save this article

The A/B testing problem with Gaussian priors

Author

Listed:
  • Azevedo, Eduardo M.
  • Mao, David
  • Montiel Olea, José Luis
  • Velez, Amilcar

Abstract

A risk-neutral firm can perform a randomized experiment (A/B test) to learn about the effects of implementing an idea of unknown quality. The firm's goal is to decide the experiment's sample size and whether or not the idea should be implemented after observing the experiment's outcome. We show that when the distribution for idea quality is Gaussian and there are linear costs of experimentation, there are exact formulae for the firm's optimal implementation decisions, the value of obtaining more data, and optimal experiment sizes. Our formulae—which assume that companies use randomized experiments to help them maximize expected profits—provide a simple alternative to i) the standard rules-of-thumb of power calculations for determining the sample size of an experiment, and also to ii) ad hoc thresholds based on statistical significance to interpret the outcome of an experiment.

Suggested Citation

  • Azevedo, Eduardo M. & Mao, David & Montiel Olea, José Luis & Velez, Amilcar, 2023. "The A/B testing problem with Gaussian priors," Journal of Economic Theory, Elsevier, vol. 210(C).
    • Handle: RePEc:eee:jetheo:v:210:y:2023:i:c:s002205312300042x
    DOI: 10.1016/j.jet.2023.105646
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S002205312300042X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jet.2023.105646?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Stoye, Jörg, 2009. "Minimax regret treatment choice with finite samples," Journal of Econometrics, Elsevier, vol. 151(1), pages 70-81, July.
    2. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    3. John List & Sally Sadoff & Mathis Wagner, 2011. "So you want to run an experiment, now what? Some simple rules of thumb for optimal experimental design," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 439-457, November.
    4. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    5. Keppo, Jussi & Moscarini, Giuseppe & Smith, Lones, 2008. "The demand for information: More heat than light," Journal of Economic Theory, Elsevier, vol. 138(1), pages 21-50, January.
    6. Charles F. Manski, 2019. "Treatment Choice With Trial Data: Statistical Decision Theory Should Supplant Hypothesis Testing," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 296-304, March.
    7. Eduardo M. Azevedo & Alex Deng & José Luis Montiel Olea & Justin Rao & E. Glen Weyl, 2020. "A/B Testing with Fat Tails," Journal of Political Economy, University of Chicago Press, vol. 128(12), pages 4614-4000.
    8. Giuseppe Moscarini & Lones Smith, 2002. "The Law of Large Demand for Information," Econometrica, Econometric Society, vol. 70(6), pages 2351-2366, November.
    9. Eduardo M. Azevedo & Alex Deng & José L. Montiel Olea & E. Glen Weyl, 2019. "Empirical Bayes Estimation of Treatment Effects with Many A/B Tests: An Overview," AEA Papers and Proceedings, American Economic Association, vol. 109, pages 43-47, May.
    10. Stoye, Jörg, 2012. "Minimax regret treatment choice with covariates or with limited validity of experiments," Journal of Econometrics, Elsevier, vol. 166(1), pages 138-156.
    11. Charles F. Manski & Aleksey Tetenov, 2019. "Trial Size for Near-Optimal Choice Between Surveillance and Aggressive Treatment: Reconsidering MSLT-II," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 305-311, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yuchen Hu & Henry Zhu & Emma Brunskill & Stefan Wager, 2024. "Minimax-Regret Sample Selection in Randomized Experiments," Papers 2403.01386, arXiv.org, revised Jun 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Manski, Charles F., 2023. "Probabilistic prediction for binary treatment choice: With focus on personalized medicine," Journal of Econometrics, Elsevier, vol. 234(2), pages 647-663.
    2. Charles F. Manski, 2021. "Econometrics for Decision Making: Building Foundations Sketched by Haavelmo and Wald," Econometrica, Econometric Society, vol. 89(6), pages 2827-2853, November.
    3. Yuchen Hu & Henry Zhu & Emma Brunskill & Stefan Wager, 2024. "Minimax-Regret Sample Selection in Randomized Experiments," Papers 2403.01386, arXiv.org, revised Jun 2024.
    4. Kitagawa, Toru & Wang, Guanyi, 2023. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," Journal of Econometrics, Elsevier, vol. 232(1), pages 109-131.
    5. Neil Christy & A. E. Kowalski, 2024. "Starting Small: Prioritizing Safety over Efficacy in Randomized Experiments Using the Exact Finite Sample Likelihood," Papers 2407.18206, arXiv.org.
    6. Charles F. Manski, 2019. "Statistical inference for statistical decisions," Papers 1909.06853, arXiv.org.
    7. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    8. Charles F. Manski & Aleksey Tetenov, 2014. "The Quantile Performance of Statistical Treatment Rules Using Hypothesis Tests to Allocate a Population to Two Treatments," CeMMAP working papers CWP44/14, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    9. Augustine Denteh & Helge Liebert, 2022. "Who Increases Emergency Department Use? New Insights from the Oregon Health Insurance Experiment," Working Papers 2201, Tulane University, Department of Economics.
    10. Charles F. Manski & Aleksey Tetenov, 2015. "Clinical trial design enabling ε-optimal treatment rules," CeMMAP working papers 60/15, Institute for Fiscal Studies.
    11. Thomas M. Russell, 2020. "Policy Transforms and Learning Optimal Policies," Papers 2012.11046, arXiv.org.
    12. Haitian Xie, 2020. "Finite-Sample Average Bid Auction," Papers 2008.10217, arXiv.org, revised Feb 2022.
    13. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised May 2024.
    14. Anders Bredahl Kock & David Preinerstorfer & Bezirgen Veliyev, 2022. "Functional Sequential Treatment Allocation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(539), pages 1311-1323, September.
    15. Davide Viviano, 2019. "Policy Targeting under Network Interference," Papers 1906.10258, arXiv.org, revised Apr 2024.
    16. Toru Kitagawa & Sokbae Lee & Chen Qiu, 2022. "Treatment Choice with Nonlinear Regret," Papers 2205.08586, arXiv.org, revised Oct 2024.
    17. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    18. Masahiro Kato & Masaaki Imaizumi & Takuya Ishihara & Toru Kitagawa, 2023. "Asymptotically Optimal Fixed-Budget Best Arm Identification with Variance-Dependent Bounds," Papers 2302.02988, arXiv.org, revised Jul 2023.
    19. Eric Mbakop & Max Tabord‐Meehan, 2021. "Model Selection for Treatment Choice: Penalized Welfare Maximization," Econometrica, Econometric Society, vol. 89(2), pages 825-848, March.
    20. Charles F. Manski, 2020. "Towards Reasonable Patient Care Under Uncertainty," Contemporary Economic Policy, Western Economic Association International, vol. 38(2), pages 227-245, April.

    More about this item

    Keywords

    Statistical Decision Theory; Optimal learning; Experiment design; A/B testing;
    All these keywords.

    JEL classification:

    • C00 - Mathematical and Quantitative Methods - - General - - - General
    • C44 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Operations Research; Statistical Decision Theory
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jetheo:v:210:y:2023:i:c:s002205312300042x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/622869 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.