IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2207.00683.html
   My bibliography  Save this paper

A Comparison of Methods for Adaptive Experimentation

Author

Listed:
  • Samantha Horn
  • Sabina J. Sloman

Abstract

We use a simulation study to compare three methods for adaptive experimentation: Thompson sampling, Tempered Thompson sampling, and Exploration sampling. We gauge the performance of each in terms of social welfare and estimation accuracy, and as a function of the number of experimental waves. We further construct a set of novel "hybrid" loss measures to identify which methods are optimal for researchers pursuing a combination of experimental aims. Our main results are: 1) the relative performance of Thompson sampling depends on the number of experimental waves, 2) Tempered Thompson sampling uniquely distributes losses across multiple experimental aims, and 3) in most cases, Exploration sampling performs similarly to random assignment.

Suggested Citation

  • Samantha Horn & Sabina J. Sloman, 2022. "A Comparison of Methods for Adaptive Experimentation," Papers 2207.00683, arXiv.org.
    • Handle: RePEc:arx:papers:2207.00683
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2207.00683
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. A. Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Shami & Alexander Teytelboym, 2020. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," CSAE Working Paper Series 2020-20, Centre for the Study of African Economies, University of Oxford.
    2. Maximilian Kasy & Anja Sautmann, 2021. "Adaptive Treatment Assignment in Experiments for Policy Choice," Econometrica, Econometric Society, vol. 89(1), pages 113-132, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bahety, Girija & Bauhoff, Sebastian & Patel, Dev & Potter, James, 2021. "Texts don’t nudge: An adaptive trial to prevent the spread of COVID-19 in India," Journal of Development Economics, Elsevier, vol. 153(C).
    2. Xiaoxue Sherry Gao & Glenn W. Harrison & Rusty Tchernis, 2020. "Behavioral Welfare Economics and Risk Preferences: A Bayesian Approach," NBER Working Papers 27685, National Bureau of Economic Research, Inc.
    3. Xiaoxue Sherry Gao & Glenn W. Harrison & Rusty Tchernis, 2023. "Behavioral welfare economics and risk preferences: a Bayesian approach," Experimental Economics, Springer;Economic Science Association, vol. 26(2), pages 273-303, April.
    4. Keisuke Hirano & Jack R. Porter, 2023. "Asymptotic Representations for Sequential Decisions, Adaptive Experiments, and Batched Bandits," Papers 2302.03117, arXiv.org.
    5. Max Cytrynbaum, 2021. "Optimal Stratification of Survey Experiments," Papers 2111.08157, arXiv.org, revised Aug 2023.
    6. Danielle Li & Lindsey R. Raymond & Peter Bergman, 2020. "Hiring as Exploration," NBER Working Papers 27736, National Bureau of Economic Research, Inc.
    7. Maximilian Kasy & Alexander Teytelboym, 2023. "Matching with semi-bandits," The Econometrics Journal, Royal Economic Society, vol. 26(1), pages 45-66.
    8. Susan Athey & Undral Byambadalai & Vitor Hadad & Sanath Kumar Krishnamurthy & Weiwen Leung & Joseph Jay Williams, 2022. "Contextual Bandits in a Survey Experiment on Charitable Giving: Within-Experiment Outcomes versus Policy Learning," Papers 2211.12004, arXiv.org.
    9. Chao Qin & Daniel Russo, 2024. "Optimizing Adaptive Experiments: A Unified Approach to Regret Minimization and Best-Arm Identification," Papers 2402.10592, arXiv.org.
    10. Masahiro Kato & Masaaki Imaizumi & Takuya Ishihara & Toru Kitagawa, 2023. "Asymptotically Optimal Fixed-Budget Best Arm Identification with Variance-Dependent Bounds," Papers 2302.02988, arXiv.org, revised Jul 2023.
    11. Michael Lechner, 2023. "Causal Machine Learning and its use for public policy," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 159(1), pages 1-15, December.
    12. Özler, Berk & Çelik, Çiğdem & Cunningham, Scott & Cuevas, P. Facundo & Parisotto, Luca, 2021. "Children on the move: Progressive redistribution of humanitarian cash transfers among refugees," Journal of Development Economics, Elsevier, vol. 153(C).
    13. Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Shami & Alexander Teytelboym, 2020. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," CESifo Working Paper Series 8535, CESifo.
    14. Arthur Charpentier & Romuald Élie & Carl Remlinger, 2023. "Reinforcement Learning in Economics and Finance," Computational Economics, Springer;Society for Computational Economics, vol. 62(1), pages 425-462, June.
    15. Kock, Anders Bredahl & Preinerstorfer, David & Veliyev, Bezirgen, 2023. "Treatment recommendation with distributional targets," Journal of Econometrics, Elsevier, vol. 234(2), pages 624-646.
    16. Martino Banchio & Giacomo Mantegazza, 2022. "Artificial Intelligence and Spontaneous Collusion," Papers 2202.05946, arXiv.org, revised Sep 2023.
    17. Sonan Memon, 2021. "Machine Learning for Economists: An Introduction," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 60(2), pages 201-211.
    18. Saskia Opitz & Dirk Sliwka & Timo Vogelsang & Tom Zimmermann, 2022. "The Targeted Assignment of Incentive Schemes," ECONtribute Discussion Papers Series 187, University of Bonn and University of Cologne, Germany.
    19. Battistin, Erich & De Nadai, Michele & Krishnan, Nandini, 2020. "The Insights and Illusions of Consumption Measurements," IZA Discussion Papers 13222, Institute of Labor Economics (IZA).
    20. Danutė Krapavickaitė, 2022. "Impact of Stratum Composition Changes on the Accuracy of the Estimates in a Sample Survey," Mathematics, MDPI, vol. 10(7), pages 1-21, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2207.00683. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.