IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v325y2025i1p118-131.html
   My bibliography  Save this article

Heteroscedasticity-aware stratified sampling to improve uplift modeling

Author

Listed:
  • Bokelmann, Björn
  • Lessmann, Stefan

Abstract

Randomized controlled trials (RCTs) are conducted in many business applications including online marketing or customer churn prevention to investigate the effect of specific treatments (coupons, retention offers, mailings, etc.). RCTs allow for the estimation of average treatment effects and the training of (uplift) models for the heterogeneity of treatment effects across individuals. The problem with RCTs is that they are costly, and this cost increases with the number of individuals included. These costs have inspired research on how to conduct experiments with a small number of individuals while still obtaining precise treatment effect estimates. We contribute to this literature a heteroskedasticity-aware stratified sampling (HS) scheme. We leverage the fact that different individuals have different noise levels in their outcome and that precise treatment effect estimation requires more observations from the “high-noise” individuals than from the “low-noise” individuals. We show theoretically and empirically that HS sampling yields significantly more precise estimates of the ATE, improves uplift models, and makes their evaluation more reliable compared to RCT data sampled completely randomly. Due to these benefits and the simplicity of our approach, we expect HS sampling to be valuable in many real-world applications in business and beyond.

Suggested Citation

  • Bokelmann, Björn & Lessmann, Stefan, 2025. "Heteroscedasticity-aware stratified sampling to improve uplift modeling," European Journal of Operational Research, Elsevier, vol. 325(1), pages 118-131.
    • Handle: RePEc:eee:ejores:v:325:y:2025:i:1:p:118-131
    DOI: 10.1016/j.ejor.2025.02.030
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221725001535
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2025.02.030?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Wright, Marvin N. & Ziegler, Andreas, 2017. "ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 77(i01).
    2. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    3. Nathan Kallus, 2018. "Optimal a priori balance in the design of controlled experiments," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(1), pages 85-112, January.
    4. Haupt, Johannes & Lessmann, Stefan, 2022. "Targeting customers under response-dependent costs," European Journal of Operational Research, Elsevier, vol. 297(1), pages 369-379.
    5. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    6. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," American Economic Review, American Economic Association, vol. 112(12), pages 3911-3940, December.
    7. Max Tabord-Meehan, 2023. "Stratification Trees for Adaptive Randomisation in Randomised Controlled Trials," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(5), pages 2646-2673.
    8. X Nie & S Wager, 2021. "Quasi-oracle estimation of heterogeneous treatment effects [TensorFlow: A system for large-scale machine learning]," Biometrika, Biometrika Trust, vol. 108(2), pages 299-319.
    9. Haupt, Johannes & Jacob, Daniel & Gubela, Robin M. & Lessmann, Stefan, 2019. "Affordable Uplift: Supervised Randomization in Controlled Exprtiments," IRTG 1792 Discussion Papers 2019-026, Humboldt University of Berlin, International Research Training Group 1792 "High Dimensional Nonstationary Time Series".
    10. Raghavendra Addanki & David Arbour & Tung Mai & Cameron Musco & Anup Rao, 2022. "Sample Constrained Treatment Effect Estimation," Papers 2210.06594, arXiv.org.
    11. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    12. Verbeke, Wouter & Olaya, Diego & Guerry, Marie-Anne & Van Belle, Jente, 2023. "To do or not to do? Cost-sensitive causal classification with individual treatment effect estimates," European Journal of Operational Research, Elsevier, vol. 305(2), pages 838-852.
    13. Aufenanger, Tobias, 2017. "Machine learning to improve experimental design," FAU Discussion Papers in Economics 16/2017, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, revised 2017.
    14. Max Cytrynbaum, 2021. "Optimal Stratification of Survey Experiments," Papers 2111.08157, arXiv.org, revised Aug 2023.
    15. Gubela, Robin M. & Lessmann, Stefan & Jaroszewicz, Szymon, 2020. "Response transformation and profit decomposition for revenue uplift modeling," European Journal of Operational Research, Elsevier, vol. 283(2), pages 647-661.
    16. Verbeke, Wouter & Dejaeger, Karel & Martens, David & Hur, Joon & Baesens, Bart, 2012. "New insights into churn prediction in the telecommunication sector: A profit driven data mining approach," European Journal of Operational Research, Elsevier, vol. 218(1), pages 211-229.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bokelmann, Björn & Lessmann, Stefan, 2024. "Improving uplift model evaluation on randomized controlled trial data," European Journal of Operational Research, Elsevier, vol. 313(2), pages 691-707.
    2. Jinglong Zhao, 2024. "Experimental Design For Causal Inference Through An Optimization Lens," Papers 2408.09607, arXiv.org, revised Aug 2024.
    3. Jiang, Liang & Phillips, Peter C.B. & Tao, Yubo & Zhang, Yichong, 2023. "Regression-adjusted estimation of quantile treatment effects under covariate-adaptive randomizations," Journal of Econometrics, Elsevier, vol. 234(2), pages 758-776.
    4. Cai, Yong & Rafi, Ahnaf, 2024. "On the performance of the Neyman Allocation with small pilots," Journal of Econometrics, Elsevier, vol. 242(1).
    5. Verbeke, Wouter & Olaya, Diego & Guerry, Marie-Anne & Van Belle, Jente, 2023. "To do or not to do? Cost-sensitive causal classification with individual treatment effect estimates," European Journal of Operational Research, Elsevier, vol. 305(2), pages 838-852.
    6. Yuehao Bai & Jizhou Liu & Max Tabord-Meehan, 2022. "Inference for Matched Tuples and Fully Blocked Factorial Designs," Papers 2206.04157, arXiv.org, revised Nov 2023.
    7. Bugni, Federico A. & Gao, Mengsi, 2023. "Inference under covariate-adaptive randomization with imperfect compliance," Journal of Econometrics, Elsevier, vol. 237(1).
    8. Yuehao Bai & Jizhou Liu & Azeem M. Shaikh & Max Tabord-Meehan, 2023. "On the Efficiency of Finely Stratified Experiments," Papers 2307.15181, arXiv.org, revised Mar 2025.
    9. Bai, Yuehao & Liu, Jizhou & Shaikh, Azeem M. & Tabord-Meehan, Max, 2024. "Inference in cluster randomized trials with matched pairs," Journal of Econometrics, Elsevier, vol. 245(1).
    10. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    11. Haupt, Johannes & Lessmann, Stefan, 2022. "Targeting customers under response-dependent costs," European Journal of Operational Research, Elsevier, vol. 297(1), pages 369-379.
    12. Laurent Davezies & Guillaume Hollard & Pedro Vergara Merino, 2024. "Revisiting Randomization with the Cube Method," Papers 2407.13613, arXiv.org, revised Jul 2025.
    13. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    14. Masahiro Kato & Masaaki Imaizumi & Takuya Ishihara & Toru Kitagawa, 2023. "Asymptotically Optimal Fixed-Budget Best Arm Identification with Variance-Dependent Bounds," Papers 2302.02988, arXiv.org, revised Jul 2023.
    15. Heinisch, Katja & Scaramella, Fabio & Schult, Christoph, 2025. "Assumption errors and forecast accuracy: A partial linear instrumental variable and double machine learning approach," IWH Discussion Papers 6/2025, Halle Institute for Economic Research (IWH).
    16. Daniel Goller, 2023. "Analysing a built-in advantage in asymmetric darts contests using causal machine learning," Annals of Operations Research, Springer, vol. 325(1), pages 649-679, June.
    17. Harsh Parikh & Trang Quynh Nguyen & Elizabeth A. Stuart & Kara E. Rudolph & Caleb H. Miles, 2025. "A Cautionary Tale on Integrating Studies with Disparate Outcome Measures for Causal Inference," Papers 2505.11014, arXiv.org.
    18. Wu, Guojun & Song, Ge & Lv, Xiaoxiang & Luo, Shikai & Shi, Chengchun & Zhu, Hongtu, 2023. "DNet: distributional network for distributional individualized treatment effects," LSE Research Online Documents on Economics 122895, London School of Economics and Political Science, LSE Library.
    19. Yiyi Huo & Yingying Fan & Fang Han, 2023. "On the adaptation of causal forests to manifold data," Papers 2311.16486, arXiv.org, revised Dec 2023.
    20. Miquel Oliu-Barton & Bary S. R. Pradelski & Nicolas Woloszko & Lionel Guetta-Jeanrenaud & Philippe Aghion & Patrick Artus & Arnaud Fontanet & Philippe Martin & Guntram B. Wolff, 2022. "The effect of COVID certificates on vaccine uptake, health outcomes, and the economy," Nature Communications, Nature, vol. 13(1), pages 1-13, December.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:325:y:2025:i:1:p:118-131. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.