Mitigating Peeking Bias and Novelty Effects in Digital Marketing Experiments An Empirical Comparison of Sequential, Stratified, and Variance-Reduction Methods

A/B testing is central to digital marketing optimization, yet the speed of decision-making often outpaces statistical discipline. This paper studies two empirically measurable threats to experimental validity on public marketing data---peeking bias from continuous monitoring and false discovery inflation under multiple comparisons---together with a third threat, novelty-driven early lifts, which is evaluated by controlled injection of temporal decay rather than by direct observation. A systematic experimental framework is proposed that layers mixture sequential probability ratio testing, CUPED-style variance reduction using pre-treatment covariates (a generalization of the original CUPED design, which requires a same-metric pre-period that is not available for every outcome here), temporal stratification with injected-novelty stress tests, and stratified subgroup screening with Benjamini-Hochberg control. The protocol is illustrated on two public marketing datasets used as proxies for broader digital experimentation rather than as native paid-search logs: a widely used public copy of the Criteo Uplift dataset (13,979,592 rows) and the Hillstrom MineThatData email campaign dataset (64,000 customers, three arms). In 1,000 simulated A/A and A/B trials on the Criteo proxy data, the sequential-testing and variance-control layers reduce realized Type-I error from 27.4% under unadjusted repeated testing to 5.1%, while lowering average sample consumed to 68.4% of the fixed-horizon budget at 81.2% power. Separately, in subgroup-screening simulations on the Hillstrom proxy data, estimated discovery precision reaches 82.3% against the held-out benchmark defined in Section 4.1. Because public datasets do not expose paid-search auction dynamics or native day-by-day novelty traces, the empirical contribution should be read as a proxy-based demonstration of digital marketing experimentation more broadly, not as a paid-search-specific validation. Even within these limits, the framework shows that lightweight statistical safeguards can preserve decision speed without sacrificing inferential validity, supporting more reliable ship decisions and more efficient experimentation budget allocation in real-world campaign optimization.