IDEAS home Printed from https://ideas.repec.org/a/inm/ormksc/v43y2024i3p590-614.html

A Bias Correction Approach for Interference in Ranking Experiments

Author

Listed:
  • Ali Goli

    (University of Washington, Seattle, Washington 98195)

  • Anja Lambrecht

    (London Business School, London NW1 4SA, United Kingdom)

  • Hema Yoganarasimhan

    (University of Washington, Seattle, Washington 98195)

Abstract

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the total average treatment effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased because of interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, that is, items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.

Suggested Citation

  • Ali Goli & Anja Lambrecht & Hema Yoganarasimhan, 2024. "A Bias Correction Approach for Interference in Ranking Experiments," Marketing Science, INFORMS, vol. 43(3), pages 590-614, May.
    • Handle: RePEc:inm:ormksc:v:43:y:2024:i:3:p:590-614
    DOI: 10.1287/mksc.2022.0046
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mksc.2022.0046
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mksc.2022.0046?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Eckles Dean & Karrer Brian & Ugander Johan, 2017. "Design and Analysis of Experiments in Networks: Reducing Bias from Interference," Journal of Causal Inference, De Gruyter, vol. 5(1), pages 1-23, March.
    2. Anindya Ghose & Panagiotis G. Ipeirotis & Beibei Li, 2014. "Examining the Impact of Ranking on Consumer Behavior and Search Engine Revenue," Management Science, INFORMS, vol. 60(7), pages 1632-1654, July.
    3. Raluca M. Ursu, 2018. "The Power of Rankings: Quantifying the Effect of Rankings on Online Consumer Search and Purchase Decisions," Marketing Science, INFORMS, vol. 37(4), pages 530-552, August.
    4. Oded Netzer & Ronen Feldman & Jacob Goldenberg & Moshe Fresko, 2012. "Mine Your Own Business: Market-Structure Surveillance Through Text Mining," Marketing Science, INFORMS, vol. 31(3), pages 521-543, May.
    5. Hema Yoganarasimhan, 2020. "Search Personalization Using Machine Learning," Management Science, INFORMS, vol. 66(3), pages 1045-1070, March.
    6. Omid Rafieian & Hema Yoganarasimhan, 2021. "Targeting and Privacy in Mobile Advertising," Marketing Science, INFORMS, vol. 40(2), pages 193-218, March.
    7. Johari, Ramesh & Li, Hannah & Weintraub, Gabriel, 2020. "Experimental Design in Two-Sided Platforms: An Analysis of Bias," Research Papers 3859, Stanford University, Graduate School of Business.
    8. David Holtz & Sinan Aral, 2020. "Limiting Bias from Test-Control Interference in Online Marketplace Experiments," Papers 2004.12162, arXiv.org.
    9. Babur De los Santos & Sergei Koulayev, 2017. "Optimizing Click-Through in Online Rankings with Endogenous Search Refinement," Marketing Science, INFORMS, vol. 36(4), pages 542-564, July.
    10. Jia Liu & Olivier Toubia & Shawndra Hill, 2021. "Content-Based Model of Web Search Behavior: An Application to TV Show Search," Management Science, INFORMS, vol. 67(10), pages 6378-6398, October.
    11. Paramveer S. Dhillon & Sinan Aral, 2021. "Modeling Dynamic User Interests: A Neural Matrix Factorization Approach," Marketing Science, INFORMS, vol. 40(6), pages 1059-1080, November.
    12. Charles F. Manski, 2013. "Identification of treatment response with social interactions," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-23, February.
    13. Hema Yoganarasimhan & Ebrahim Barzegary & Abhishek Pani, 2023. "Design and Evaluation of Optimal Free Trials," Management Science, INFORMS, vol. 69(6), pages 3220-3240, June.
    14. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    15. David Holtz & Ruben Lobel & Inessa Liskovich & Sinan Aral, 2020. "Reducing Interference Bias in Online Marketplace Pricing Experiments," Papers 2004.12489, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lu Liu & Jiaming Fang & Zhenyu Ji, 2026. "Posture as information: Streamer postures and the purchase of micro and small enterprise resellers in livestreaming," Journal of the Academy of Marketing Science, Springer, vol. 54(1), pages 254-280, February.
    2. Ali Goli & David H. Reiley & Hongkai Zhang, 2025. "Personalizing Ad Load to Optimize Subscription and Ad Revenues: Product Strategies Constructed from Experiments on Pandora," Marketing Science, INFORMS, vol. 44(2), pages 327-352, March.
    3. Zikun Ye & Zhiqi Zhang & Dennis J. Zhang & Heng Zhang & Renyu Zhang, 2026. "Deep Learning-Based Causal Inference for Large-Scale Combinatorial Experiments: Theory and Empirical Evidence," Management Science, INFORMS, vol. 72(7), pages 5611-5634, July.
    4. repec:arx:papers:2411.16552 is not listed on IDEAS

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wei Zhou & Mingfeng Lin & Mo Xiao & Lu Fang, 2025. "Higher Precision Is Not Always Better: Search Algorithm and Consumer Engagement," Management Science, INFORMS, vol. 71(7), pages 6204-6226, July.
    2. Yuyan Wang & Long Tao & Xian Xing Zhang, 2025. "Recommending for a Multi-Sided Marketplace: A Multi-Objective Hierarchical Approach," Marketing Science, INFORMS, vol. 44(1), pages 1-29, January.
    3. Iavor Bojinov & David Simchi-Levi & Jinglong Zhao, 2023. "Design and Analysis of Switchback Experiments," Management Science, INFORMS, vol. 69(7), pages 3759-3777, July.
    4. Shengjun Mao & Sanjeev Dewan & Yi-Jen (Ian) Ho, 2023. "Personalized Ranking at a Mobile App Distribution Platform," Information Systems Research, INFORMS, vol. 34(3), pages 811-827, September.
    5. Hannah Li & Geng Zhao & Ramesh Johari & Gabriel Y. Weintraub, 2021. "Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms," Papers 2104.12222, arXiv.org.
    6. Honka, Elisabeth & Seiler, Stephan & Ursu, Raluca, 2024. "Consumer search: What can we learn from pre-purchase data?," Journal of Retailing, Elsevier, vol. 100(1), pages 114-129.
    7. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    8. Wei Zhou & Zidong Wang, 2020. "Competing for Search Traffic in Query Markets: Entry Strategy, Platform Design, and Entrepreneurship," Working Papers 20-12, NET Institute.
    9. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    10. Tianshu Sun & Zhe Yuan & Chunxiao Li & Kaifu Zhang & Jun Xu, 2024. "The Value of Personal Data in Internet Commerce: A High-Stakes Field Experiment on Data Regulation Policy," Management Science, INFORMS, vol. 70(4), pages 2645-2660, April.
    11. Giovanni Compiani & Gregory Lewis & Sida Peng & Peichun Wang, 2024. "Online Search and Optimal Product Rankings: An Empirical Framework," Marketing Science, INFORMS, vol. 43(3), pages 615-636, May.
    12. Rafael P. Greminger, 2022. "Optimal Search and Discovery," Management Science, INFORMS, vol. 68(5), pages 3904-3924, May.
    13. Ozan Candogan & Chen Chen & Rad Niazadeh, 2024. "Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization," Management Science, INFORMS, vol. 70(6), pages 4069-4086, June.
    14. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Sciences Po Economics Publications (main) hal-03455978, HAL.
    15. Chenshuo Sun, 2025. "How Does Prepopulating Search Bars with Keywords Affect Online Consumer Behavior? A Field Experiment," Marketing Science, INFORMS, vol. 44(6), pages 1217-1231, November.
    16. Ata Jameei Osgouei & Andrew T. Ching & Brian T. Ratchford & Shervin Shahrokhi Tehrani, 2026. "Estimating Position and Social Influence Effects in Online Search," Marketing Science, INFORMS, vol. 45(1), pages 203-223, January.
    17. Ariel Boyarsky & Hongseok Namkoong & Jean Pouget-Abadie, 2023. "Modeling Interference Using Experiment Roll-out," Papers 2305.10728, arXiv.org, revised Aug 2023.
    18. Chris Gu & Yike Wang, 2022. "Consumer Online Search with Partially Revealed Information," Management Science, INFORMS, vol. 68(6), pages 4215-4235, June.
    19. Gu, Chris & Wang, Yike, 2022. "Consumer online search with partially revealed information," LSE Research Online Documents on Economics 109871, London School of Economics and Political Science, LSE Library.
    20. Amelia Fletcher & Peter L Ormosi & Rahul Savani, 2023. "Recommender Systems and Supplier Competition on Platforms," Journal of Competition Law and Economics, Oxford University Press, vol. 19(3), pages 397-426.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormksc:v:43:y:2024:i:3:p:590-614. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.