IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2306.00485.html

Some searches may not work properly. We apologize for the inconvenience.

   My bibliography  Save this paper

Causal Estimation of User Learning in Personalized Systems

Author

Listed:
  • Evan Munro
  • David Jones
  • Jennifer Brennan
  • Roland Nelet
  • Vahab Mirrokni
  • Jean Pouget-Abadie

Abstract

In online platforms, the impact of a treatment on an observed outcome may change over time as 1) users learn about the intervention, and 2) the system personalization, such as individualized recommendations, change over time. We introduce a non-parametric causal model of user actions in a personalized system. We show that the Cookie-Cookie-Day (CCD) experiment, designed for the measurement of the user learning effect, is biased when there is personalization. We derive new experimental designs that intervene in the personalization system to generate the variation necessary to separately identify the causal effect mediated through user learning and personalization. Making parametric assumptions allows for the estimation of long-term causal effects based on medium-term experiments. In simulations, we show that our new designs successfully recover the dynamic causal effects of interest.

Suggested Citation

  • Evan Munro & David Jones & Jennifer Brennan & Roland Nelet & Vahab Mirrokni & Jean Pouget-Abadie, 2023. "Causal Estimation of User Learning in Personalized Systems," Papers 2306.00485, arXiv.org.
    • Handle: RePEc:arx:papers:2306.00485
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2306.00485
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    2. Iavor Bojinov & Neil Shephard, 2019. "Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(528), pages 1665-1682, October.
    3. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    4. Patrick Bajari & Brian Burdick & Guido W. Imbens & Lorenzo Masoero & James McQueen & Thomas Richardson & Ido M. Rosen, 2021. "Multiple Randomization Designs," Papers 2112.13495, arXiv.org.
    5. Yuchen Hu & Stefan Wager, 2022. "Switchback Experiments under Geometric Mixing," Papers 2209.00197, arXiv.org, revised Apr 2024.
    6. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    7. Ramesh Johari & Hannah Li & Inessa Liskovich & Gabriel Y. Weintraub, 2022. "Experimental Design in Two-Sided Platforms: An Analysis of Bias," Management Science, INFORMS, vol. 68(10), pages 7069-7089, October.
    8. Michael P. Leung, 2021. "Rate-Optimal Cluster-Randomized Designs for Spatial Interference," Papers 2111.04219, arXiv.org, revised Sep 2022.
    9. Tyler J. VanderWeele & Eric J. Tchetgen Tchetgen, 2017. "Mediation analysis with time varying exposures and mediators," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(3), pages 917-938, June.
    10. Kosuke Imai & Dustin Tingley & Teppei Yamamoto, 2013. "Experimental designs for identifying causal mechanisms," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(1), pages 5-51, January.
    11. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    12. Tyler J. Vanderweele, 2011. "Controlled Direct and Mediated Effects: Definition, Identification and Bounds," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 38(3), pages 551-563, September.
    13. Iavor Bojinov & Ashesh Rambachan & Neil Shephard, 2021. "Panel experiments and dynamic causal effects: A finite population perspective," Quantitative Economics, Econometric Society, vol. 12(4), pages 1171-1196, November.
    14. repec:cup:cbooks:9780521885881 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Jingjing Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Luofeng Liao & Christian Kroer, 2024. "Statistical Inference and A/B Testing in Fisher Markets and Paced Auctions," Papers 2406.15522, arXiv.org, revised Aug 2024.
    2. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    3. Ruohan Zhan & Shichao Han & Yuchen Hu & Zhenling Jiang, 2024. "Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach," Papers 2406.14380, arXiv.org, revised Jul 2024.
    4. Nian Si, 2023. "Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach," Papers 2310.17496, arXiv.org, revised Apr 2024.
    5. Ruoxuan Xiong & Alex Chin & Sean J. Taylor, 2024. "Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs," Papers 2406.06768, arXiv.org.
    6. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    7. Ozan Candogan & Chen Chen & Rad Niazadeh, 2024. "Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization," Management Science, INFORMS, vol. 70(6), pages 4069-4086, June.
    8. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Jingjing Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org.
    9. Christopher Harshaw & Fredrik Savje & Yitan Wang, 2022. "A Design-Based Riesz Representation Framework for Randomized Experiments," Papers 2210.08698, arXiv.org, revised Oct 2022.
    10. Supriya Tiwari & Pallavi Basu, 2024. "Quasi-randomization tests for network interference," Papers 2403.16673, arXiv.org.
    11. Acharya, Avidit & Blackwell, Matthew & Sen, Maya, 2016. "Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects," American Political Science Review, Cambridge University Press, vol. 110(3), pages 512-529, August.
    12. Michael P. Leung, 2023. "Design of Cluster-Randomized Trials with Cross-Cluster Interference," Papers 2310.18836, arXiv.org, revised Nov 2023.
    13. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    14. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    15. Kara E. Rudolph & Iván Díaz, 2022. "When the ends do not justify the means: Learning who is predicted to have harmful indirect effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(S2), pages 573-589, December.
    16. Zhaonan Qu & Ruoxuan Xiong & Jizhou Liu & Guido Imbens, 2021. "Semiparametric Estimation of Treatment Effects in Observational Studies with Heterogeneous Partial Interference," Papers 2107.12420, arXiv.org, revised Jun 2024.
    17. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    18. Shaina J. Alexandria & Michael G. Hudgens & Allison E. Aiello, 2023. "Assessing intervention effects in a randomized trial within a social network," Biometrics, The International Biometric Society, vol. 79(2), pages 1409-1419, June.
    19. Vivek F. Farias & Andrew A. Li & Tianyi Peng & Andrew Zheng, 2022. "Markovian Interference in Experiments," Papers 2206.02371, arXiv.org, revised Jun 2022.
    20. Georgia Papadogeorgou & Kosuke Imai & Jason Lyall & Fan Li, 2022. "Causal inference with spatio‐temporal data: Estimating the effects of airstrikes on insurgent violence in Iraq," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(5), pages 1969-1999, November.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2306.00485. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.