IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v68y2022i10p7069-7089.html
   My bibliography  Save this article

Experimental Design in Two-Sided Platforms: An Analysis of Bias

Author

Listed:
  • Ramesh Johari

    (Management Science and Engineering, Stanford University, Stanford, California 94305)

  • Hannah Li

    (Management Science and Engineering, Stanford University, Stanford, California 94305)

  • Inessa Liskovich

    (Airbnb Inc., San Francisco, California 94117)

  • Gabriel Y. Weintraub

    (Graduate School of Business, Stanford University, Stanford, California 94305)

Abstract

We develop an analytical framework to study experimental design in two-sided marketplaces. Many of these experiments exhibit interference , where an intervention applied to one market participant influences the behavior of another participant. This interference leads to biased estimates of the treatment effect of the intervention. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side “customer” randomization ( CR ) and supply-side “listing” randomization ( LR ), along with their associated estimators. We show that good experimental design depends on market balance; in highly demand-constrained markets, CR is unbiased, whereas LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, whereas CR is biased. We also introduce and study a novel experimental design based on two-sided randomization ( TSR ) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance while yielding relatively low bias in intermediate regimes of market balance.

Suggested Citation

  • Ramesh Johari & Hannah Li & Inessa Liskovich & Gabriel Y. Weintraub, 2022. "Experimental Design in Two-Sided Platforms: An Analysis of Bias," Management Science, INFORMS, vol. 68(10), pages 7069-7089, October.
    • Handle: RePEc:inm:ormnsc:v:68:y:2022:i:10:p:7069-7089
    DOI: 10.1287/mnsc.2021.4247
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.2021.4247
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.2021.4247?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Patrick Bajari & Brian Burdick & Guido W. Imbens & Lorenzo Masoero & James McQueen & Thomas Richardson & Ido M. Rosen, 2021. "Multiple Randomization Designs," Papers 2112.13495, arXiv.org.
    2. Charles F. Manski, 2013. "Identification of treatment response with social interactions," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-23, February.
    3. G W Basse & A Feller & P Toulis, 2019. "Randomization tests of causal effects under interference," Biometrika, Biometrika Trust, vol. 106(2), pages 487-494.
    4. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    5. David Holtz & Ruben Lobel & Inessa Liskovich & Sinan Aral, 2020. "Reducing Interference Bias in Online Marketplace Pricing Experiments," Papers 2004.12489, arXiv.org.
    6. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ruoxuan Xiong & Alex Chin & Sean J. Taylor, 2024. "Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs," Papers 2406.06768, arXiv.org.
    2. Ruohan Zhan & Shichao Han & Yuchen Hu & Zhenling Jiang, 2024. "Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach," Papers 2406.14380, arXiv.org, revised Jul 2024.
    3. Evan Munro & David Jones & Jennifer Brennan & Roland Nelet & Vahab Mirrokni & Jean Pouget-Abadie, 2023. "Causal Estimation of User Learning in Personalized Systems," Papers 2306.00485, arXiv.org.
    4. Luofeng Liao & Christian Kroer, 2024. "Statistical Inference and A/B Testing in Fisher Markets and Paced Auctions," Papers 2406.15522, arXiv.org, revised Mar 2025.
    5. Xinqi Chen & Xingyu Bai & Zeyu Zheng & Nian Si, 2025. "Bias Analysis of Experiments for Multi-Item Multi-Period Inventory Control Policies," Papers 2501.11996, arXiv.org.
    6. Anish Agarwal & Sarah H. Cen & Devavrat Shah & Christina Lee Yu, 2022. "Network Synthetic Interventions: A Causal Framework for Panel Data Under Network Interference," Papers 2210.11355, arXiv.org, revised Oct 2023.
    7. Ido Bright & Arthur Delarue & Ilan Lobel, 2022. "Reducing Marketplace Interference Bias Via Shadow Prices," Papers 2205.02274, arXiv.org, revised Mar 2024.
    8. Nathan Kallus, 2023. "Treatment Effect Risk: Bounds and Inference," Management Science, INFORMS, vol. 69(8), pages 4579-4590, August.
    9. Shan Huang & Chen Wang & Yuan Yuan & Jinglong Zhao & Brocco & Zhang, 2023. "Estimating Effects of Long-Term Treatments," Papers 2308.08152, arXiv.org, revised Dec 2024.
    10. Mohsen Bayati & Yuwei Luo & William Overman & Sadegh Shirani & Ruoxuan Xiong, 2024. "Higher-Order Causal Message Passing for Experimentation with Complex Interference," Papers 2411.00945, arXiv.org, revised Feb 2025.
    11. Yifan Wu & Ramesh Johari & Vasilis Syrgkanis & Gabriel Y. Weintraub, 2024. "Switchback Price Experiments with Forward-Looking Demand," Papers 2410.14904, arXiv.org.
    12. Luo, Shikai & Yang, Ying & Shi, Chengchun & Yao, Fang & Ye, Jieping & Zhu, Hongtu, 2024. "Policy evaluation for temporal and/or spatial dependent experiments," LSE Research Online Documents on Economics 122741, London School of Economics and Political Science, LSE Library.
    13. Luofeng Liao & Christian Kroer, 2024. "Bootstrapping Fisher Market Equilibrium and First-Price Pacing Equilibrium," Papers 2402.02303, arXiv.org, revised Mar 2025.
    14. Luofeng Liao & Christian Kroer, 2023. "Statistical Inference and A/B Testing for First-Price Pacing Equilibria," Papers 2301.02276, arXiv.org, revised Jun 2023.
    15. Shuze Chen & David Simchi-Levi & Chonghuan Wang, 2024. "Experimenting on Markov Decision Processes with Local Treatments," Papers 2407.19618, arXiv.org, revised Oct 2024.
    16. Steven Wilkins Reeves & Shane Lubold & Arun G. Chandrasekhar & Tyler H. McCormick, 2024. "Model-Based Inference and Experimental Design for Interference Using Partial Network Data," Papers 2406.11940, arXiv.org.
    17. Ozan Candogan & Chen Chen & Rad Niazadeh, 2024. "Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization," Management Science, INFORMS, vol. 70(6), pages 4069-4086, June.
    18. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hannah Li & Geng Zhao & Ramesh Johari & Gabriel Y. Weintraub, 2021. "Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms," Papers 2104.12222, arXiv.org.
    2. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jan 2025.
    3. Han, Kevin & Basse, Guillaume & Bojinov, Iavor, 2024. "Population interference in panel experiments," Journal of Econometrics, Elsevier, vol. 238(1).
    4. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    5. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    6. Tadao Hoshino & Takahide Yanagi, 2024. "Causal Inference with Noncompliance and Unknown Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(548), pages 2869-2880, October.
    7. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    8. Stefan Wager & Kuang Xu, 2019. "Experimenting in Equilibrium," Papers 1903.02124, arXiv.org, revised Jun 2020.
    9. Ramesh Johari & Hannah Li & Inessa Liskovich & Gabriel Weintraub, 2020. "Experimental Design in Two-Sided Platforms: An Analysis of Bias," Papers 2002.05670, arXiv.org, revised Sep 2021.
    10. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    11. Chabé-Ferret, Sylvain & Reynaud, Arnaud & Tène, Eva, 2021. "Water Quality, Policy Diffusion Effects and Farmers’ Behavior," TSE Working Papers 21-1229, Toulouse School of Economics (TSE).
    12. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    13. Tadao Hoshino & Takahide Yanagi, 2023. "Randomization Test for the Specification of Interference Structure," Papers 2301.05580, arXiv.org, revised Dec 2023.
    14. Vivek F. Farias & Andrew A. Li & Tianyi Peng & Andrew Zheng, 2022. "Markovian Interference in Experiments," Papers 2206.02371, arXiv.org, revised Jun 2022.
    15. Vazquez-Bare, Gonzalo, 2023. "Identification and estimation of spillover effects in randomized experiments," Journal of Econometrics, Elsevier, vol. 237(1).
    16. Nicolas Debarsy & Julie Le Gallo, 2025. "Identification of spatial spillovers: Do’s and don'ts," Post-Print hal-04549691, HAL.
    17. Ali Goli & Anja Lambrecht & Hema Yoganarasimhan, 2024. "A Bias Correction Approach for Interference in Ranking Experiments," Marketing Science, INFORMS, vol. 43(3), pages 590-614, May.
    18. Kitagawa, Toru & Wang, Guanyi, 2023. "Who should get vaccinated? Individualized allocation of vaccines over SIR network," Journal of Econometrics, Elsevier, vol. 232(1), pages 109-131.
    19. Anish Agarwal & Sarah H. Cen & Devavrat Shah & Christina Lee Yu, 2022. "Network Synthetic Interventions: A Causal Framework for Panel Data Under Network Interference," Papers 2210.11355, arXiv.org, revised Oct 2023.
    20. Davide Viviano, 2020. "Experimental Design under Network Interference," Papers 2003.08421, arXiv.org, revised Jul 2022.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:68:y:2022:i:10:p:7069-7089. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.