IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1903.11198.html

Parallel Experimentation and Competitive Interference on Online Advertising Platforms

Author

Listed:
  • Caio Waisman
  • Navdeep S. Sahni
  • Harikesh S. Nair
  • Xiliang Lin

Abstract

This paper studies the measurement of advertising effects on online platforms when parallel experimentation occurs, that is, when multiple advertisers experiment concurrently. It provides a framework that makes precise how parallel experimentation affects the experiment's value: while ignoring parallel experimentation yields an estimate of the average effect of advertising in-place, which has limited value in decision-making in an environment with variable advertising competition, accounting for parallel experimentation captures the actual uncertainty advertisers face due to competitive actions. It then implements an experimental design that enables the estimation of these effects on JD.com, a large e-commerce platform that is also a publisher of digital ads. Using traditional and kernel-based estimators, it shows that not accounting for competitive actions can result in the advertiser inaccurately estimating the advertising lift by a factor of two or higher, which can be consequential for decision-making.

Suggested Citation

  • Caio Waisman & Navdeep S. Sahni & Harikesh S. Nair & Xiliang Lin, 2019. "Parallel Experimentation and Competitive Interference on Online Advertising Platforms," Papers 1903.11198, arXiv.org, revised Feb 2024.
    • Handle: RePEc:arx:papers:1903.11198
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1903.11198
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Daron Acemoglu, 2010. "Theory, General Equilibrium, and Political Economy in Development Economics," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 17-32, Summer.
    2. Burke, Raymond R & Srull, Thomas K, 1988. "Competitive Interference and Consumer Memory for Advertising," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 15(1), pages 55-68, June.
    3. Duncan Simester & Artem Timoshenko & Spyros I. Zoumpoulis, 2020. "Efficiently Evaluating Targeting Policies: Improving on Champion vs. Challenger Experiments," Management Science, INFORMS, vol. 66(8), pages 3412-3424, August.
    4. Navdeep S. Sahni, 2015. "Erratum to: Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 249-250, September.
    5. Randall Lewis & David Reiley, 2014. "Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!," Quantitative Marketing and Economics (QME), Springer, vol. 12(3), pages 235-266, September.
    6. Bradley T. Shapiro, 2018. "Positive Spillovers and Free Riding in Advertising of Prescription Pharmaceuticals: The Case of Antidepressants," Journal of Political Economy, University of Chicago Press, vol. 126(1), pages 381-437.
    7. QI Li & Desheng Ouyang & Jeffrey S. Racine, 2013. "Categorical semiparametric varying‐coefficient models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(4), pages 551-579, June.
    8. Avi Goldfarb & Catherine Tucker, 2011. "Search Engine Advertising: Channel Substitution When Pricing Ads to Context," Management Science, INFORMS, vol. 57(3), pages 458-470, March.
    9. Li, Qi & Racine, Jeff, 2003. "Nonparametric estimation of distributions with categorical and continuous data," Journal of Multivariate Analysis, Elsevier, vol. 86(2), pages 266-292, August.
    10. Navdeep S Sahni & Harikesh S Nair, 2020. "Does Advertising Serve as a Signal? Evidence from a Field Experiment in Mobile Search," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(3), pages 1529-1564.
    11. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    12. Navdeep S. Sahni, 2015. "Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 203-247, September.
    13. Jean-Pierre Dubé & Günter Hitsch & Puneet Manchanda, 2005. "An Empirical Model of Advertising Dynamics," Quantitative Marketing and Economics (QME), Springer, vol. 3(2), pages 107-144, June.
    14. Randall Lewis & Dan Nguyen, 2015. "Display advertising’s competitive spillovers to consumer search," Quantitative Marketing and Economics (QME), Springer, vol. 13(2), pages 93-115, June.
    15. Susan Athey & Dean Eckles & Guido W. Imbens, 2018. "Exact p-Values for Network Interference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 230-240, January.
    16. Brett R. Gordon & Florian Zettelmeyer & Neha Bhargava & Dan Chapsky, 2019. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," Marketing Science, INFORMS, vol. 38(2), pages 193-225, March.
    17. Hudgens, Michael G. & Halloran, M. Elizabeth, 2008. "Toward Causal Inference With Interference," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 832-842, June.
    18. Stefan Wager & Kuang Xu, 2021. "Experimenting in Equilibrium," Management Science, INFORMS, vol. 67(11), pages 6694-6715, November.
    19. Sridhar Narayanan & Kirthi Kalyanam, 2015. "Position Effects in Search Advertising and their Moderators: A Regression Discontinuity Approach," Marketing Science, INFORMS, vol. 34(3), pages 388-407, May.
    20. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    21. Keller, Kevin Lane, 1991. "Memory and Evaluation Effects in Competitive Advertising Environments," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 17(4), pages 463-476, March.
    22. Racine, Jeff & Li, Qi, 2004. "Nonparametric estimation of regression functions with both categorical and continuous data," Journal of Econometrics, Elsevier, vol. 119(1), pages 99-130, March.
    23. Navdeep Sahni, 2015. "Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 203-247, September.
    24. Thomas Blake & Chris Nosko & Steven Tadelis, 2015. "Consumer Heterogeneity and Paid Search Effectiveness: A Large‐Scale Field Experiment," Econometrica, Econometric Society, vol. 83, pages 155-174, January.
    25. Keller, Kevin Lane, 1987. "Memory Factors in Advertising: The Effect of Advertising Retrieval Cues on Brand Evaluations," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 14(3), pages 316-333, December.
    26. Naufel J. Vilcassim & Vrinda Kadiyali & Pradeep K. Chintagunta, 1999. "Investigating Dynamic Multifirm Market Interactions in Price and Advertising," Management Science, INFORMS, vol. 45(4), pages 499-518, April.
    27. James Heckman & Neil Hohmann & Jeffrey Smith & Michael Khoo, 2000. "Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(2), pages 651-694.
    28. Andrey Simonov & Shawndra Hill, 2021. "Competitive Advertising on Brand Search: Traffic Stealing and Click Quality," Marketing Science, INFORMS, vol. 40(5), pages 923-945, September.
    29. Jean-Pierre Dubé & Puneet Manchanda, 2005. "Differences in Dynamic Brand Competition Across Markets: An Empirical Analysis," Marketing Science, INFORMS, vol. 24(1), pages 81-95, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Brett R Gordon & Kinshuk Jerath & Zsolt Katona & Sridhar Narayanan & Jiwoong Shin & Kenneth C Wilbur, 2019. "Inefficiencies in Digital Advertising Markets," Papers 1912.09012, arXiv.org, revised Feb 2020.
    2. Sarah Moshary, 2025. "Does Sponsored Search Advertising Augment Organic Search? Evidence from an E-Commerce Platform," Management Science, INFORMS, vol. 71(11), pages 9687-9709, November.
    3. Geng, Tong & Lin, Xiliang & Nair, Harikesh S. & Hao, Jun & Xiang, Bin & Fan, Shurui, 2020. "Comparison Lift: Bandit-Based Experimentation System for Online Advertising," Research Papers 3904, Stanford University, Graduate School of Business.
    4. Caio Waisman & Brett R. Gordon, 2023. "Multicell experiments for marginal treatment effect estimation of digital ads," Papers 2302.13857, arXiv.org, revised Apr 2025.
    5. Garrett Johnson & Julian Runge & Eric Seufert, 2022. "Privacy-Centric Digital Advertising: Implications for Research," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 9(1), pages 49-54, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Navdeep S. Sahni & Harikesh S. Nair, 2020. "Sponsorship Disclosure and Consumer Deception: Experimental Evidence from Native Advertising in Mobile Search," Marketing Science, INFORMS, vol. 39(1), pages 5-32, January.
    2. Brett R Gordon & Kinshuk Jerath & Zsolt Katona & Sridhar Narayanan & Jiwoong Shin & Kenneth C Wilbur, 2019. "Inefficiencies in Digital Advertising Markets," Papers 1912.09012, arXiv.org, revised Feb 2020.
    3. Michael Thomas & Marcel Goic & Kirthi Kalyanam, 2025. "Long Lags and Large Returns: Experimental Evidence from Advertising to Businesses," Management Science, INFORMS, vol. 71(10), pages 8750-8766, October.
    4. Weijia Dai & Hyunjin Kim & Michael Luca, 2023. "Frontiers: Which Firms Gain from Digital Advertising? Evidence from a Field Experiment," Marketing Science, INFORMS, vol. 42(3), pages 429-439, May.
    5. Kirthi Kalyanam & John McAteer & Jonathan Marek & James Hodges & Lifeng Lin, 2018. "Cross channel effects of search engine advertising on brick & mortar retail sales: Meta analysis of large scale field experiments on Google.com," Quantitative Marketing and Economics (QME), Springer, vol. 16(1), pages 1-42, March.
    6. Brett R. Gordon & Robert Moakler & Florian Zettelmeyer, 2023. "Predicted Incrementality by Experimentation (PIE) for Ad Measurement," Papers 2304.06828, arXiv.org, revised Apr 2026.
    7. Ali Goli & Jason Huang & David Reiley & Nickolai M. Riabov, 2025. "Measuring consumer sensitivity to audio advertising: a long-run field experiment on Pandora internet radio," Quantitative Marketing and Economics (QME), Springer, vol. 23(3), pages 347-377, September.
    8. Hana Choi & Carl F. Mela & Santiago R. Balseiro & Adam Leary, 2020. "Online Display Advertising Markets: A Literature Review and Future Directions," Information Systems Research, INFORMS, vol. 31(2), pages 556-575, June.
    9. Garrett A. Johnson & Randall A. Lewis & David H. Reiley, 2017. "When Less Is More: Data and Power in Advertising Experiments," Marketing Science, INFORMS, vol. 36(1), pages 43-53, January.
    10. Navdeep Sahni, 2015. "Effect of temporal spacing between advertising exposures: Evidence from online field experiments," Quantitative Marketing and Economics (QME), Springer, vol. 13(3), pages 203-247, September.
    11. Raluca Mihaela Ursu & Andrey Simonov & Eunkyung An, 2025. "Online Advertising as Passive Search," Management Science, INFORMS, vol. 71(2), pages 1050-1073, February.
    12. Chen He & Tobias J. Klein, 2023. "Advertising as a Reminder: Evidence from the Dutch State Lottery," Marketing Science, INFORMS, vol. 42(5), pages 892-909, September.
    13. Ruichang Lu & Qiaowei Shen & Tenghui Wang & Xiaojun Zhang, 2022. "Frenemies: Corporate Advertising Under Common Ownership," Management Science, INFORMS, vol. 68(6), pages 4645-4669, June.
    14. Li, Xiaolin & Singh Rao, Raghunath & Narasimhan, Om & Gao, Xing, 2022. "Stay positive or go negative? Memory imperfections and messaging strategy," LSE Research Online Documents on Economics 113556, London School of Economics and Political Science, LSE Library.
    15. Brett R. Gordon & Florian Zettelmeyer & Neha Bhargava & Dan Chapsky, 2019. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," Marketing Science, INFORMS, vol. 38(2), pages 193-225, March.
    16. Stephan Seiler & Song Yao & Wenbo Wang, 2017. "Does Online Word of Mouth Increase Demand? (And How?) Evidence from a Natural Experiment," Marketing Science, INFORMS, vol. 36(6), pages 838-861, November.
    17. Johannes Hermle & Giorgio Martini, 2022. "Valid and Unobtrusive Measurement of Returns to Advertising through Asymmetric Budget Split," Papers 2207.00206, arXiv.org.
    18. Ali Goli & Jason Huang & David Reiley & Nickolai M. Riabov, 2024. "Measuring Consumer Sensitivity to Audio Advertising: A Long-Run Field Experiment on Pandora Internet Radio," Papers 2412.05516, arXiv.org.
    19. Thomas W. Frick & Rodrigo Belo & Rahul Telang, 2023. "Incentive Misalignments in Programmatic Advertising: Evidence from a Randomized Field Experiment," Management Science, INFORMS, vol. 69(3), pages 1665-1686, March.
    20. Navdeep S. Sahni & Dan Zou & Pradeep K. Chintagunta, 2017. "Do Targeted Discount Offers Serve as Advertising? Evidence from 70 Field Experiments," Management Science, INFORMS, vol. 63(8), pages 2688-2705, August.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1903.11198. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.