IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2306.00485.html
   My bibliography  Save this paper

Causal Estimation of User Learning in Personalized Systems

Author

Listed:
  • Evan Munro
  • David Jones
  • Jennifer Brennan
  • Roland Nelet
  • Vahab Mirrokni
  • Jean Pouget-Abadie

Abstract

In online platforms, the impact of a treatment on an observed outcome may change over time as 1) users learn about the intervention, and 2) the system personalization, such as individualized recommendations, change over time. We introduce a non-parametric causal model of user actions in a personalized system. We show that the Cookie-Cookie-Day (CCD) experiment, designed for the measurement of the user learning effect, is biased when there is personalization. We derive new experimental designs that intervene in the personalization system to generate the variation necessary to separately identify the causal effect mediated through user learning and personalization. Making parametric assumptions allows for the estimation of long-term causal effects based on medium-term experiments. In simulations, we show that our new designs successfully recover the dynamic causal effects of interest.

Suggested Citation

  • Evan Munro & David Jones & Jennifer Brennan & Roland Nelet & Vahab Mirrokni & Jean Pouget-Abadie, 2023. "Causal Estimation of User Learning in Personalized Systems," Papers 2306.00485, arXiv.org.
    • Handle: RePEc:arx:papers:2306.00485
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2306.00485
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    2. Iavor Bojinov & Neil Shephard, 2019. "Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(528), pages 1665-1682, October.
    3. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    4. Patrick Bajari & Brian Burdick & Guido W. Imbens & Lorenzo Masoero & James McQueen & Thomas Richardson & Ido M. Rosen, 2021. "Multiple Randomization Designs," Papers 2112.13495, arXiv.org.
    5. Yuchen Hu & Stefan Wager, 2022. "Switchback Experiments under Geometric Mixing," Papers 2209.00197, arXiv.org, revised Apr 2024.
    6. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    7. Ramesh Johari & Hannah Li & Inessa Liskovich & Gabriel Y. Weintraub, 2022. "Experimental Design in Two-Sided Platforms: An Analysis of Bias," Management Science, INFORMS, vol. 68(10), pages 7069-7089, October.
    8. Michael P. Leung, 2021. "Rate-Optimal Cluster-Randomized Designs for Spatial Interference," Papers 2111.04219, arXiv.org, revised Sep 2022.
    9. Tyler J. VanderWeele & Eric J. Tchetgen Tchetgen, 2017. "Mediation analysis with time varying exposures and mediators," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(3), pages 917-938, June.
    10. Kosuke Imai & Dustin Tingley & Teppei Yamamoto, 2013. "Experimental designs for identifying causal mechanisms," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(1), pages 5-51, January.
    11. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    12. Tyler J. Vanderweele, 2011. "Controlled Direct and Mediated Effects: Definition, Identification and Bounds," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 38(3), pages 551-563, September.
    13. Iavor Bojinov & Ashesh Rambachan & Neil Shephard, 2021. "Panel experiments and dynamic causal effects: A finite population perspective," Quantitative Economics, Econometric Society, vol. 12(4), pages 1171-1196, November.
    14. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Jingjing Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    2. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Jingjing Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org.
    3. Nian Si, 2023. "Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach," Papers 2310.17496, arXiv.org, revised Apr 2024.
    4. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    5. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Mar 2024.
    6. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    7. Michael P. Leung, 2022. "Causal Inference Under Approximate Neighborhood Interference," Econometrica, Econometric Society, vol. 90(1), pages 267-293, January.
    8. Luofeng Liao & Christian Kroer, 2023. "Statistical Inference and A/B Testing for First-Price Pacing Equilibria," Papers 2301.02276, arXiv.org, revised Jun 2023.
    9. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    10. Davide Viviano, 2020. "Experimental Design under Network Interference," Papers 2003.08421, arXiv.org, revised Jul 2022.
    11. Shi, Chengchun & Wan, Runzhe & Song, Ge & Luo, Shikai & Zhu, Hongtu & Song, Rui, 2023. "A multiagent reinforcement learning framework for off-policy evaluation in two-sided markets," LSE Research Online Documents on Economics 117174, London School of Economics and Political Science, LSE Library.
    12. Guillaume W Basse & Edoardo M Airoldi, 2018. "Model-assisted design of experiments in the presence of network-correlated outcomes," Biometrika, Biometrika Trust, vol. 105(4), pages 849-858.
    13. Stefan Wager & Kuang Xu, 2019. "Experimenting in Equilibrium," Papers 1903.02124, arXiv.org, revised Jun 2020.
    14. Christopher Harshaw & Fredrik Savje & Yitan Wang, 2022. "A Design-Based Riesz Representation Framework for Randomized Experiments," Papers 2210.08698, arXiv.org, revised Oct 2022.
    15. Gonzalo Vazquez-Bare, 2017. "Identification and Estimation of Spillover Effects in Randomized Experiments," Papers 1711.02745, arXiv.org, revised Jan 2022.
    16. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    17. Acharya, Avidit & Blackwell, Matthew & Sen, Maya, 2016. "Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects," American Political Science Review, Cambridge University Press, vol. 110(3), pages 512-529, August.
    18. Hao, Shiming, 2021. "True structure change, spurious treatment effect? A novel approach to disentangle treatment effects from structure changes," MPRA Paper 108679, University Library of Munich, Germany.
    19. Michael P. Leung, 2023. "Design of Cluster-Randomized Trials with Cross-Cluster Interference," Papers 2310.18836, arXiv.org, revised Nov 2023.
    20. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2306.00485. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.