IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2605.23706.html

Algorithm or Creative? A Three-Arm Experimental Design for Decomposing Algorithmic Bias in Platform A/B Tests

Author

Listed:
  • Pallavi Pal
  • Anjana Susarla

Abstract

Online advertising platforms host hundreds of thousands of A/B tests, but the platform's delivery algorithm routes each creative to the audience it predicts will engage. Every two-arm test therefore conflates the creative's effect with the algorithm's targeting response, and adjusting for the realized audience is biased because audience is a post-treatment mediator. We propose a three-arm design that adds an arm exposing the algorithm to the treatment metadata while holding the user-facing creative identical to control, point-identifying the natural indirect (algorithmic) and direct (creative) effects without sequential ignorability. In a live Meta campaign with a women-targeted text fragment, the algorithmic channel raises female impression share by +2.07 ppt while the creative channel moves it by -0.68 ppt; roughly three-quarters of the absolute reallocation is algorithmic, and a conventional two-arm test understates the algorithmic channel by a factor of two. The design isolates the contribution of platform's algorithm to the outcome which is separable from creative content.

Suggested Citation

  • Pallavi Pal & Anjana Susarla, 2026. "Algorithm or Creative? A Three-Arm Experimental Design for Decomposing Algorithmic Bias in Platform A/B Tests," Papers 2605.23706, arXiv.org.
    • Handle: RePEc:arx:papers:2605.23706
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2605.23706
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gordon Burtch & Robert Moakler & Brett R. Gordon & Poppy Zhang & Shawndra Hill, 2025. "Characterizing and Minimizing Divergent Delivery in Meta Advertising Experiments," Papers 2508.21251, arXiv.org.
    2. Donald P. Green & Shang E. Ha & John G. Bullock, 2010. "Enough Already about “Black Box†Experiments: Studying Mediation Is More Difficult than Most Scholars Suppose," The ANNALS of the American Academy of Political and Social Science, , vol. 628(1), pages 200-208, March.
    3. Jens Ludwig & Jeffrey R. Kling & Sendhil Mullainathan, 2011. "Mechanism Experiments and Policy Evaluations," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 17-38, Summer.
    4. Davidson, Russell & Flachaire, Emmanuel, 2008. "The wild bootstrap, tamed at last," Journal of Econometrics, Elsevier, vol. 146(1), pages 162-169, September.
    5. Brett R. Gordon & Florian Zettelmeyer & Neha Bhargava & Dan Chapsky, 2019. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," Marketing Science, INFORMS, vol. 38(2), pages 193-225, March.
    6. Kosuke Imai & Dustin Tingley & Teppei Yamamoto, 2013. "Experimental designs for identifying causal mechanisms," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(1), pages 5-51, January.
    7. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    8. Acharya, Avidit & Blackwell, Matthew & Sen, Maya, 2016. "Explaining Causal Findings Without Bias: Detecting and Assessing Direct Effects," American Political Science Review, Cambridge University Press, vol. 110(3), pages 512-529, August.
    9. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    10. Sinan Aral & Dylan Walker, 2011. "Creating Social Contagion Through Viral Product Design: A Randomized Trial of Peer Influence in Networks," Management Science, INFORMS, vol. 57(9), pages 1623-1639, February.
    11. Anja Lambrecht & Catherine Tucker, 2019. "Algorithmic Bias? An Empirical Study of Apparent Gender-Based Discrimination in the Display of STEM Career Ads," Management Science, INFORMS, vol. 65(7), pages 2966-2981, July.
    12. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    13. MacKinnon, James G. & White, Halbert, 1985. "Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties," Journal of Econometrics, Elsevier, vol. 29(3), pages 305-325, September.
    14. James J. Heckman & Rodrigo Pinto, 2015. "Econometric Mediation Analyses: Identifying the Sources of Treatment Effects from Experimentally Estimated Production Technologies with Unmeasured and Mismeasured Inputs," Econometric Reviews, Taylor & Francis Journals, vol. 34(1-2), pages 6-31, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    2. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2023. "Fast and reliable jackknife and bootstrap methods for cluster‐robust inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 671-694, August.
    3. Bruce E. Hansen, 2025. "Standard Errors for Difference‐in‐Difference Regression," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(3), pages 291-309, April.
    4. Sunny R. Karim & Morten {O}rregaard Nielsen & James G. MacKinnon & Matthew D. Webb, 2026. "Improved Inference for CSDID Using the Cluster Jackknife," Papers 2602.12043, arXiv.org.
    5. Tom Boot & Gianmaria Niccodemi & Tom Wansbeek, 2023. "Unbiased estimation of the OLS covariance matrix when the errors are clustered," Empirical Economics, Springer, vol. 64(6), pages 2511-2533, June.
    6. James G. MacKinnon, 2025. "When can we trust cluster-robust inference?," Canadian Stata Users' Group Meetings 2025 11, Stata Users Group.
    7. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    8. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    9. MacKinnon, James G. & Webb, Matthew D., 2017. "Pitfalls when Estimating Treatment Effects Using Clustered Data," Queen's Economics Department Working Papers 274713, Queen's University - Department of Economics.
    10. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    11. James G. MacKinnon, 2020. "Wild cluster bootstrap confidence intervals," L'Actualité Economique, Société Canadienne de Science Economique, vol. 96(4), pages 721-743.
    12. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    13. David Roodman & James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2019. "Fast and wild: Bootstrap inference in Stata using boottest," Stata Journal, StataCorp LLC, vol. 19(1), pages 4-60, March.
    14. Sviták, Jan & Tichem, Jan & Haasbeek, Stefan, 2021. "Price effects of search advertising restrictions," International Journal of Industrial Organization, Elsevier, vol. 77(C).
    15. David J. Price, 2026. "Power Law Heteroskedasticity," Working Papers tecipa-822, University of Toronto, Department of Economics.
    16. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 106, University of California, Davis, Department of Economics.
    17. Dykstra, Holly & Fernández Guerrico, Sofía, 2026. "Offsetting the Earnings Disincentive in Public Housing: Evidence from a Behaviorally Informed Field Intervention," IZA Discussion Papers 18483, IZA Network @ LISER.
    18. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    19. Packalen, Mikko & Wirjanto, Tony S., 2012. "Inference about clustering and parametric assumptions in covariance matrix estimation," Computational Statistics & Data Analysis, Elsevier, vol. 56(1), pages 1-14, January.
    20. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2605.23706. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.