IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/10098.html
   My bibliography  Save this paper

Adaptive Experiments for Policy Choice : Phone Calls for Home Reading in Kenya

Author

Listed:
  • Esposito Acosta,Bruno Nicola
  • Sautmann,Anja

Abstract

Adaptive sampling in experiments with multiple waves can improve learning for “policy choiceproblems” where the goal is to select the optimal intervention or treatment among several options. This paperuses a real-world policy choice problem to demonstrate the advantages of adaptive sampling and propose solutions tocommon issues in applying the method. The application is a test of six formats for automated calls to parents in Kenyathat encourage reading with children at home. The adaptive ‘exploration sampling’ algorithm is used to efficientlyidentify the call with the highest rate of engagement. Simulations show that adaptive sampling increased theposterior probability of the chosen arm being optimal from 86 to 93 percent and more than halved the posterior expected regret. The paper discusses a range of implementationaspects, including how to decide about research design parameters such as the number of experimental waves.

Suggested Citation

  • Esposito Acosta,Bruno Nicola & Sautmann,Anja, 2022. "Adaptive Experiments for Policy Choice : Phone Calls for Home Reading in Kenya," Policy Research Working Paper Series 10098, The World Bank.
    • Handle: RePEc:wbk:wbrwps:10098
    as

    Download full text from publisher

    File URL: http://documents.worldbank.org/curated/en/099337506232226851/pdf/IDU078c6dff10259d0409d0997f0d753a7e2cb72.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    4. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    5. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    6. Rodriguez-Segura, Daniel & Campton, Cole & Crouch, Luis & Slade, Timothy S., 2021. "Looking beyond changes in averages in evaluating foundational learning: Some inequality measures," International Journal of Educational Development, Elsevier, vol. 84(C).
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    8. Peter Bergman & Eric W. Chan, 2021. "Leveraging Parents through Low-Cost Technology: The Impact of High-Frequency Information on Student Achievement," Journal of Human Resources, University of Wisconsin Press, vol. 56(1), pages 125-158.
    9. Bahety, Girija & Bauhoff, Sebastian & Patel, Dev & Potter, James, 2021. "Texts don’t nudge: An adaptive trial to prevent the spread of COVID-19 in India," Journal of Development Economics, Elsevier, vol. 153(C).
    10. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    11. Peter Bergman, 2021. "Parent-Child Information Frictions and Human Capital Investment: Evidence from a Field Experiment," Journal of Political Economy, University of Chicago Press, vol. 129(1), pages 286-322.
    12. A. Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Shami & Alexander Teytelboym, 2020. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," CSAE Working Paper Series 2020-20, Centre for the Study of African Economies, University of Oxford.
    13. Benjamin N. York & Susanna Loeb & Christopher Doss, 2019. "One Step at a Time: The Effects of an Early Literacy Text-Messaging Program for Parents of Preschoolers," Journal of Human Resources, University of Wisconsin Press, vol. 54(3), pages 537-566.
    14. Samuel Berlinski & Matias Busso & Taryn Dinkelman & Claudia Martínez A., 2021. "Reducing Parent-School Information Gaps and Improving Education Outcomes: Evidence from High-Frequency Text Messages," NBER Working Papers 28581, National Bureau of Economic Research, Inc.
    15. Matthew A. Kraft & Manuel Monti-Nussbaum, 2017. "Can Schools Enable Parents to Prevent Summer Learning Loss? A Text-Messaging Field Experiment to Promote Literacy Skills," The ANNALS of the American Academy of Political and Social Science, , vol. 674(1), pages 85-112, November.
    16. Kasy, Maximilian, 2016. "Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead," Political Analysis, Cambridge University Press, vol. 24(3), pages 324-338, July.
    17. Susan E. Mayer & Ariel Kalil & Philip Oreopoulos & Sebastian Gallegos, 2019. "Using Behavioral Insights to Increase Parental Engagement: The Parents and Children Together Intervention," Journal of Human Resources, University of Wisconsin Press, vol. 54(4), pages 900-925.
    18. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Mar 2023.
    19. Frederico Finan & Demian Pouzo, 2022. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," NBER Working Papers 29756, National Bureau of Economic Research, Inc.
    20. Eric Bettinger & Nina Cunha & Guilherme Lichand & Ricardo Madeira, 2020. "Are the effects of informational interventions driven by salience?," ECON - Working Papers 350, Department of Economics - University of Zurich, revised May 2021.
    21. Daniel Russo, 2020. "Simple Bayesian Algorithms for Best-Arm Identification," Operations Research, INFORMS, vol. 68(6), pages 1625-1647, November.
    22. Christopher Doss & Erin M. Fahle & Susanna Loeb & Benjamin N. York, 2019. "More Than Just a Nudge: Supporting Kindergarten Parents with Differentiated and Personalized Text Messages," Journal of Human Resources, University of Wisconsin Press, vol. 54(3), pages 567-603.
    23. Abhijit V. Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2020. "A Theory of Experimenters: Robustness, Randomization, and Balance," American Economic Review, American Economic Association, vol. 110(4), pages 1206-1230, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Noam Angrist & Peter Bergman & Moitshepi Matsheng, 2022. "Experimental evidence on learning using low-tech when school is out," Nature Human Behaviour, Nature, vol. 6(7), pages 941-950, July.
    2. Maldonado, Joana Elisa & De Witte, Kristof, 2021. "The impact of information provision to parents: Experimental evidence on student outcomes," Journal of Behavioral and Experimental Finance, Elsevier, vol. 31(C).
    3. Hermes, Henning & Lergetporer, Philipp & Peter, Frauke & Wiederhold, Simon, 2021. "Behavioral Barriers and the Socioeconomic Gap in Child Care Enrollment," IZA Discussion Papers 14698, Institute of Labor Economics (IZA).
    4. Noam Angrist & Peter Bergman & Moitshepi Matsheng, 2020. "School’s Out: Experimental Evidence on Limiting Learning Loss Using “Low-Tech” in a Pandemic," NBER Working Papers 28205, National Bureau of Economic Research, Inc.
    5. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    6. Karthik Muralidharan & Mauricio Romero & Kaspar Wüthrich, 2019. "Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments," NBER Working Papers 26562, National Bureau of Economic Research, Inc.
    7. Simon Calmar Andersen & Ulrik Hvidman, 2020. "Implementing Educational Interventions at Scale," Working Papers 2020-039, Human Capital and Economic Opportunity Working Group.
    8. Amaral, Sofia & Dinarte-Diaz, Lelys & Dominguez, Patricio & Perez-Vincent, Santiago M., 2024. "Helping families help themselves: The (Un)intended impacts of a digital parenting program," Journal of Development Economics, Elsevier, vol. 166(C).
    9. Agostinelli, Francesco & Avitabile, Ciro & Bobba, Matteo, 2021. "Enhancing Human Capital in Children: A Case Study on Scaling," TSE Working Papers 21-1196, Toulouse School of Economics (TSE), revised Oct 2023.
    10. Joana Elisa Maldonado & Kristof De Witte & Koen Declercq, 2022. "The effects of parental involvement in homework: two randomised controlled trials in financial education," Empirical Economics, Springer, vol. 62(3), pages 1439-1464, March.
    11. Hashibul Hassan & Asad Islam & Abu Siddique & Liang Choon Wang, 2021. "Telementoring and homeschooling during school closures: A randomized experiment in rural Bangladesh," Munich Papers in Political Economy 13, Munich School of Politics and Public Policy and the School of Management at the Technical University of Munich.
    12. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    13. Andersen, Simon Calmar & Guul, Thorbjørn Sejr & Humlum, Maria Knoth, 2019. "Instruction Time, Information, and Student Achievement: Evidence from a Field Experiment," IZA Discussion Papers 12543, Institute of Labor Economics (IZA).
    14. Nicoletti, Cheti & Sevilla, Almudena & Tonei, Valentina, 2022. "Gender Stereotypes in the Family," IZA Discussion Papers 15773, Institute of Labor Economics (IZA).
    15. Beam, Emily A. & Mukherjee, Priya & Navarro-Sola, Laia, 2022. "Lowering Barriers to Remote Education: Experimental Impacts on Parental Responses and Learning," IZA Discussion Papers 15596, Institute of Labor Economics (IZA).
    16. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    17. Allen, James & Mahumane, Arlete & Riddell, James & Rosenblat, Tanya & Yang, Dean & Yu, Hang, 2022. "Teaching and incentives: Substitutes or complements?," Economics of Education Review, Elsevier, vol. 91(C).
    18. Naik, Gopal & Chitre, Chetan & Bhalla, Manaswini & Rajan, Jothsna, 2020. "Impact of use of technology on student learning outcomes: Evidence from a large-scale experiment in India," World Development, Elsevier, vol. 127(C).
    19. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    20. Eble, Alex & Frost, Chris & Camara, Alpha & Bouy, Baboucarr & Bah, Momodou & Sivaraman, Maitri & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Brady, Tony & Gawron, Piotr & Vansteelandt, Stijn & Boone, P, 2021. "How much can we remedy very low learning levels in rural parts of low-income countries? Impact and generalizability of a multi-pronged para-teacher intervention from a cluster-randomized trial in the ," Journal of Development Economics, Elsevier, vol. 148(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:10098. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Roula I. Yazigi (email available below). General contact details of provider: https://edirc.repec.org/data/dvewbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.