IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2004.12162.html
   My bibliography  Save this paper

Limiting Bias from Test-Control Interference in Online Marketplace Experiments

Author

Listed:
  • David Holtz
  • Sinan Aral

Abstract

In an A/B test, the typical objective is to measure the total average treatment effect (TATE), which measures the difference between the average outcome if all users were treated and the average outcome if all users were untreated. However, a simple difference-in-means estimator will give a biased estimate of the TATE when outcomes of control units depend on the outcomes of treatment units, an issue we refer to as test-control interference. Using a simulation built on top of data from Airbnb, this paper considers the use of methods from the network interference literature for online marketplace experimentation. We model the marketplace as a network in which an edge exists between two sellers if their goods substitute for one another. We then simulate seller outcomes, specifically considering a "status quo" context and "treatment" context that forces all sellers to lower their prices. We use the same simulation framework to approximate TATE distributions produced by using blocked graph cluster randomization, exposure modeling, and the Hajek estimator for the difference in means. We find that while blocked graph cluster randomization reduces the bias of the naive difference-in-means estimator by as much as 62%, it also significantly increases the variance of the estimator. On the other hand, the use of more sophisticated estimators produces mixed results. While some provide (small) additional reductions in bias and small reductions in variance, others lead to increased bias and variance. Overall, our results suggest that experiment design and analysis techniques from the network experimentation literature are promising tools for reducing bias due to test-control interference in marketplace experiments.

Suggested Citation

  • David Holtz & Sinan Aral, 2020. "Limiting Bias from Test-Control Interference in Online Marketplace Experiments," Papers 2004.12162, arXiv.org.
    • Handle: RePEc:arx:papers:2004.12162
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2004.12162
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    2. Charles F. Manski, 2013. "Identification of treatment response with social interactions," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-23, February.
    3. Moore, Ryan T., 2012. "Multivariate Continuous Blocking to Improve Political Science Experiments," Political Analysis, Cambridge University Press, vol. 20(4), pages 460-479.
    4. Vasant Dhar & Tomer Geva & Gal Oestreicher-Singer & Arun Sundararajan, 2014. "Prediction in Economic Networks," Information Systems Research, INFORMS, vol. 25(2), pages 264-284, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ravi Bapna & Edward McFowland & Probal Mojumder & Jui Ramaprasad & Akhmed Umyarov, 2023. "So, Who Likes You? Evidence from a Randomized Field Experiment," Management Science, INFORMS, vol. 69(7), pages 3939-3957, July.
    2. Ugander Johan & Yin Hao, 2023. "Randomized graph cluster randomization," Journal of Causal Inference, De Gruyter, vol. 11(1), pages 1-53, January.
    3. Hannah Li & Geng Zhao & Ramesh Johari & Gabriel Y. Weintraub, 2021. "Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms," Papers 2104.12222, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Supriya Tiwari & Pallavi Basu, 2024. "Quasi-randomization tests for network interference," Papers 2403.16673, arXiv.org.
    2. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    3. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    4. Michael P. Leung, 2021. "Rate-Optimal Cluster-Randomized Designs for Spatial Interference," Papers 2111.04219, arXiv.org, revised Sep 2022.
    5. Vivek F. Farias & Andrew A. Li & Tianyi Peng & Andrew Zheng, 2022. "Markovian Interference in Experiments," Papers 2206.02371, arXiv.org, revised Jun 2022.
    6. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    7. Davide Viviano & Jess Rudder, 2020. "Policy design in experiments with unknown interference," Papers 2011.08174, arXiv.org, revised Dec 2023.
    8. Davide Viviano & Lihua Lei & Guido Imbens & Brian Karrer & Okke Schrijvers & Liang Shi, 2023. "Causal clustering: design of cluster experiments under network interference," Papers 2310.14983, arXiv.org, revised Jan 2024.
    9. Anish Agarwal & Sarah H. Cen & Devavrat Shah & Christina Lee Yu, 2022. "Network Synthetic Interventions: A Causal Framework for Panel Data Under Network Interference," Papers 2210.11355, arXiv.org, revised Oct 2023.
    10. Davide Viviano, 2020. "Experimental Design under Network Interference," Papers 2003.08421, arXiv.org, revised Jul 2022.
    11. David Holtz & Ruben Lobel & Inessa Liskovich & Sinan Aral, 2020. "Reducing Interference Bias in Online Marketplace Pricing Experiments," Papers 2004.12489, arXiv.org.
    12. Guillaume W Basse & Edoardo M Airoldi, 2018. "Model-assisted design of experiments in the presence of network-correlated outcomes," Biometrika, Biometrika Trust, vol. 105(4), pages 849-858.
    13. C. Tort`u & I. Crimaldi & F. Mealli & L. Forastiere, 2020. "Modelling Network Interference with Multi-valued Treatments: the Causal Effect of Immigration Policy on Crime Rates," Papers 2003.10525, arXiv.org, revised Jun 2020.
    14. Stefan Wager & Kuang Xu, 2019. "Experimenting in Equilibrium," Papers 1903.02124, arXiv.org, revised Jun 2020.
    15. Fredrik Savje, 2021. "Causal inference with misspecified exposure mappings: separating definitions and assumptions," Papers 2103.06471, arXiv.org, revised Mar 2023.
    16. Yuchen Hu & Shuangning Li & Stefan Wager, 2021. "Average Direct and Indirect Causal Effects under Interference," Papers 2104.03802, arXiv.org, revised Jan 2022.
    17. Aufenanger, Tobias, 2017. "Machine learning to improve experimental design," FAU Discussion Papers in Economics 16/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, revised 2017.
    18. Yann Bramoullé & Habiba Djebbari & Bernard Fortin, 2020. "Peer Effects in Networks: A Survey," Annual Review of Economics, Annual Reviews, vol. 12(1), pages 603-629, August.
    19. Giovanni Cerulli, 2014. "ntreatreg: a Stata module for estimation of treatment effects in the presence of neighborhood interactions," United Kingdom Stata Users' Group Meetings 2014 15, Stata Users Group.
    20. Gibbons, Steve & Overman, Henry G. & Patacchini, Eleonora, 2015. "Spatial Methods," Handbook of Regional and Urban Economics, in: Gilles Duranton & J. V. Henderson & William C. Strange (ed.), Handbook of Regional and Urban Economics, edition 1, volume 5, chapter 0, pages 115-168, Elsevier.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2004.12162. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.