IDEAS home Printed from https://ideas.repec.org/p/ecl/stabus/3904.html
   My bibliography  Save this paper

Comparison Lift: Bandit-Based Experimentation System for Online Advertising

Author

Listed:
  • Geng, Tong

    (JD.com)

  • Lin, Xiliang

    (JD.com)

  • Nair, Harikesh S.

    (JD.com and Stanford U)

  • Hao, Jun

    (JD.com)

  • Xiang, Bin

    (JD.com)

  • Fan, Shurui

    (JD.com)

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are twofold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.

Suggested Citation

  • Geng, Tong & Lin, Xiliang & Nair, Harikesh S. & Hao, Jun & Xiang, Bin & Fan, Shurui, 2020. "Comparison Lift: Bandit-Based Experimentation System for Online Advertising," Research Papers 3904, Stanford University, Graduate School of Business.
    • Handle: RePEc:ecl:stabus:3904
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2009.07899
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Caio Waisman & Harikesh S. Nair & Carlos Carrion, 2025. "Online Causal Inference for Advertising in Real-Time Bidding Auctions," Marketing Science, INFORMS, vol. 44(1), pages 176-195, January.
    2. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(1), pages 45-97.
    3. John Rust, 2019. "Has Dynamic Programming Improved Decision Making?," Annual Review of Economics, Annual Reviews, vol. 11(1), pages 833-858, August.
    4. Caio Waisman & Navdeep S. Sahni & Harikesh S. Nair & Xiliang Lin, 2025. "Parallel Experimentation and Competitive Interference on Online Advertising Platforms," Marketing Science, INFORMS, vol. 44(2), pages 437-456, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brett R Gordon & Kinshuk Jerath & Zsolt Katona & Sridhar Narayanan & Jiwoong Shin & Kenneth C Wilbur, 2019. "Inefficiencies in Digital Advertising Markets," Papers 1912.09012, arXiv.org, revised Feb 2020.
    2. François Gardes, 2021. "Endogenous Prices in a Riemannian Geometry Framework," Post-Print halshs-03325414, HAL.
    3. Ali Hortacsu & Olivia R. Natan & Hayden Parsley & Timothy Schwieg & Kevin R. Williams, 2021. "Organizational Structure and Pricing: Evidence from a Large U.S. Airline," Cowles Foundation Discussion Papers 2312R3, Cowles Foundation for Research in Economics, Yale University, revised Jan 2023.
    4. Herrera Gómez, Marcos & Ruiz Marín, Manuel & Mur Lacambra, Jesús, 2014. "Testing Spatial Causality in Cross-section Data," MPRA Paper 56678, University Library of Munich, Germany.
    5. Omar Al-Ubaydli & John A. List, 2019. "How natural field experiments have enhanced our understanding of unemployment," Nature Human Behaviour, Nature, vol. 3(1), pages 33-39, January.
    6. Bernard Fortin & Nicolas Jacquemet & Bruce Shearer, 2008. "Policy Analysis in Health-Services Market: Accounting for Quality and Quantity," Annals of Economics and Statistics, GENES, issue 91-92, pages 293-319.
    7. Leeper, Eric M. & Zha, Tao, 2003. "Modest policy interventions," Journal of Monetary Economics, Elsevier, vol. 50(8), pages 1673-1700, November.
    8. Pesaran, M.H. & Smith, L.V. & Smith, R.P, 2005. "What if the UK has Joined the Euro in 1999? An Empirical Evaluation using a Global VAR," Cambridge Working Papers in Economics 0528, Faculty of Economics, University of Cambridge.
    9. Battistin, Erich & Chesher, Andrew, 2014. "Treatment effect estimation with covariate measurement error," Journal of Econometrics, Elsevier, vol. 178(2), pages 707-715.
    10. Sonia Bhalotra & Samantha Rawlings, 2013. "Gradients of the Intergenerational Transmission of Health in Developing Countries," The Review of Economics and Statistics, MIT Press, vol. 95(2), pages 660-672, May.
    11. Bhalotra, Sonia R. & Umana-Aponte, Marcela, 2010. "The Dynamics of Women's Labour Supply in Developing Countries," IZA Discussion Papers 4879, Institute of Labor Economics (IZA).
    12. Coleman, Stephen, 2005. "Testing Theories with Qualitative and Quantitative Predictions," MPRA Paper 105171, University Library of Munich, Germany.
    13. Moffitt, Robert A., 2002. "Welfare programs and labor supply," Handbook of Public Economics, in: A. J. Auerbach & M. Feldstein (ed.), Handbook of Public Economics, edition 1, volume 4, chapter 34, pages 2393-2430, Elsevier.
    14. Sabatini, Fabio, 2006. "The Empirics of Social Capital and Economic Development: A Critical Perspective," Knowledge, Technology, Human Capital Working Papers 12097, Fondazione Eni Enrico Mattei (FEEM).
    15. Jean-Sébastien Lenfant, 2021. "Eugen (Evgeny Evgenievich) Slutsky (1880-1948)," Working Papers hal-03628273, HAL.
    16. Peter Adams & Michael D. Hurd & Daniel L. McFadden & Angela Merrill & Tiago Ribeiro, 2004. "Healthy, Wealthy, and Wise? Tests for Direct Causal Paths between Health and Socioeconomic Status," NBER Chapters, in: Perspectives on the Economics of Aging, pages 415-526, National Bureau of Economic Research, Inc.
    17. Sandeep Devanatha Pillai & Brent Goldfarb & David Kirsch, 2024. "Lovely and likely: Using historical methods to improve inference to the best explanation in strategy," Strategic Management Journal, Wiley Blackwell, vol. 45(8), pages 1539-1566, August.
    18. Ray C. Fair, 2019. "Some Important Macro Points," Cowles Foundation Discussion Papers 2165R, Cowles Foundation for Research in Economics, Yale University, revised Jul 2019.
    19. Atella, Vincenzo & Braione, Manuela & Ferrara, Giancarlo & Resce, Giuliano, 2023. "Cohesion Policy Funds and local government autonomy: Evidence from Italian municipalities," Socio-Economic Planning Sciences, Elsevier, vol. 87(PB).
    20. Smirnova, Natalia, 2003. "Job search behavior of unemployed in Russia," BOFIT Discussion Papers 13/2003, Bank of Finland Institute for Emerging Economies (BOFIT).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:stabus:3904. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/gsstaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.