IDEAS home Printed from https://ideas.repec.org/p/ecl/stabus/3904.html
   My bibliography  Save this paper

Comparison Lift: Bandit-Based Experimentation System for Online Advertising

Author

Listed:
  • Geng, Tong

    (JD.com)

  • Lin, Xiliang

    (JD.com)

  • Nair, Harikesh S.

    (JD.com and Stanford U)

  • Hao, Jun

    (JD.com)

  • Xiang, Bin

    (JD.com)

  • Fan, Shurui

    (JD.com)

Abstract

Comparison Lift is an experimentation-as-a-service (EaaS) application for testing online advertising audiences and creatives at JD.com. Unlike many other EaaS tools that focus primarily on fixed sample A/B testing, Comparison Lift deploys a custom bandit-based experimentation algorithm. The advantages of the bandit-based approach are twofold. First, it aligns the randomization induced in the test with the advertiser's goals from testing. Second, by adapting experimental design to information acquired during the test, it reduces substantially the cost of experimentation to the advertiser. Since launch in May 2019, Comparison Lift has been utilized in over 1,500 experiments. We estimate that utilization of the product has helped increase click-through rates of participating advertising campaigns by 46% on average. We estimate that the adaptive design in the product has generated 27% more clicks on average during testing compared to a fixed sample A/B design. Both suggest significant value generation and cost savings to advertisers from the product.

Suggested Citation

  • Geng, Tong & Lin, Xiliang & Nair, Harikesh S. & Hao, Jun & Xiang, Bin & Fan, Shurui, 2020. "Comparison Lift: Bandit-Based Experimentation System for Online Advertising," Research Papers 3904, Stanford University, Graduate School of Business.
    • Handle: RePEc:ecl:stabus:3904
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2009.07899
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Caio Waisman & Harikesh S. Nair & Carlos Carrion, 2019. "Online Causal Inference for Advertising in Real-Time Bidding Auctions," Papers 1908.08600, arXiv.org, revised Feb 2024.
    2. Caio Waisman & Navdeep S. Sahni & Harikesh S. Nair & Xiliang Lin, 2019. "Parallel Experimentation and Competitive Interference on Online Advertising Platforms," Papers 1903.11198, arXiv.org, revised Feb 2024.
    3. John Rust, 2019. "Has Dynamic Programming Improved Decision Making?," Annual Review of Economics, Annual Reviews, vol. 11(1), pages 833-858, August.
    4. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(1), pages 45-97.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brett R Gordon & Kinshuk Jerath & Zsolt Katona & Sridhar Narayanan & Jiwoong Shin & Kenneth C Wilbur, 2019. "Inefficiencies in Digital Advertising Markets," Papers 1912.09012, arXiv.org, revised Feb 2020.
    2. François Gardes, 2021. "Endogenous Prices in a Riemannian Geometry Framework," Post-Print halshs-03325414, HAL.
    3. Fabio Sabatini, 2006. "Social Capital and Labour Productivity in Italy," Working Papers 2006.30, Fondazione Eni Enrico Mattei.
    4. Herrera Gómez, Marcos & Ruiz Marín, Manuel & Mur Lacambra, Jesús, 2014. "Testing Spatial Causality in Cross-section Data," MPRA Paper 56678, University Library of Munich, Germany.
    5. Omar Al-Ubaydli & John A. List, 2019. "How natural field experiments have enhanced our understanding of unemployment," Nature Human Behaviour, Nature, vol. 3(1), pages 33-39, January.
    6. Bernard Fortin & Nicolas Jacquemet & Bruce Shearer, 2008. "Policy Analysis in Health-Services Market: Accounting for Quality and Quantity," Annals of Economics and Statistics, GENES, issue 91-92, pages 293-319.
    7. Leeper, Eric M. & Zha, Tao, 2003. "Modest policy interventions," Journal of Monetary Economics, Elsevier, vol. 50(8), pages 1673-1700, November.
    8. Baktash, Mehrzad B. & Heywood, John S. & Jirjahn, Uwe, 2022. "Worker stress and performance pay: German survey evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 201(C), pages 276-291.
    9. Oliver Falck & Michael Fritsch & Stephan Heblich, 2009. "Bohemians, Human Capital, and Regional Economic Growth," Jena Economics Research Papers 2009-049, Friedrich-Schiller-University Jena.
    10. Antonio Filippin & Luca Nunziata, 2019. "Monetary effects of inequality: lessons from the euro experiment," The Journal of Economic Inequality, Springer;Society for the Study of Economic Inequality, vol. 17(2), pages 99-124, June.
    11. Pesaran, M.H. & Smith, L.V. & Smith, R.P, 2005. "What if the UK has Joined the Euro in 1999? An Empirical Evaluation using a Global VAR," Cambridge Working Papers in Economics 0528, Faculty of Economics, University of Cambridge.
    12. Battistin, Erich & Chesher, Andrew, 2014. "Treatment effect estimation with covariate measurement error," Journal of Econometrics, Elsevier, vol. 178(2), pages 707-715.
    13. Sonia Bhalotra & Samantha Rawlings, 2013. "Gradients of the Intergenerational Transmission of Health in Developing Countries," The Review of Economics and Statistics, MIT Press, vol. 95(2), pages 660-672, May.
    14. Caron, E.A.M. & Daniels, H.A.M., 2008. "Explanation of exceptional values in multi-dimensional business databases," European Journal of Operational Research, Elsevier, vol. 188(3), pages 884-897, August.
    15. Bhalotra, Sonia R. & Umana-Aponte, Marcela, 2010. "The Dynamics of Women's Labour Supply in Developing Countries," IZA Discussion Papers 4879, Institute of Labor Economics (IZA).
    16. repec:zbw:bofitp:2003_013 is not listed on IDEAS
    17. Coleman, Stephen, 2005. "Testing Theories with Qualitative and Quantitative Predictions," MPRA Paper 105171, University Library of Munich, Germany.
    18. Omar Al-Ubaydli & John List, 2013. "On the Generalizability of Experimental Results in Economics: With A Response To Camerer," Artefactual Field Experiments j0001, The Field Experiments Website.
    19. Moffitt, Robert A., 2002. "Welfare programs and labor supply," Handbook of Public Economics, in: A. J. Auerbach & M. Feldstein (ed.), Handbook of Public Economics, edition 1, volume 4, chapter 34, pages 2393-2430, Elsevier.
    20. massimo franchi, 2002. "A Non-Causal Identification Scheme for Vector Autoregressions," Computing in Economics and Finance 2002 290, Society for Computational Economics.
    21. Vollebergh, Herman R.J. & Melenberg, Bertrand & Dijkgraaf, Elbert, 2009. "Identifying reduced-form relations with panel data: The case of pollution and income," Journal of Environmental Economics and Management, Elsevier, vol. 58(1), pages 27-42, July.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:stabus:3904. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/gsstaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.