STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments

TL;DR

STATE addresses the robustness gap in variance reduction for online controlled experiments with heavy-tailed metrics by fusing ML-based regression adjustment with a Student's $t$-distributed residual via a variational EM framework. It extends to ratio metrics through a consistent linear transformation that preserves unbiased estimation and variance properties, enabling robust variance reduction beyond Gaussian assumptions. Empirical results on synthetic data and Meituan real data show STATE achieves substantial variance reductions—often exceeding $50\%$ relative to state-of-the-art baselines—and can deliver the same statistical power with roughly half the observations. The approach is particularly advantageous when outliers and tail heaviness are pronounced, improving practical experiment efficiency and decision-making speed in large-scale online platforms.

Abstract

Online controlled experiments play a crucial role in enabling data-driven decisions across a wide range of companies. Variance reduction is an effective technique to improve the sensitivity of experiments, achieving higher statistical power while using fewer samples and shorter experimental periods. However, typical variance reduction methods (e.g., regression-adjusted estimators) are built upon the intuitional assumption of Gaussian distributions and cannot properly characterize the real business metrics with heavy-tailed distributions. Furthermore, outliers diminish the correlation between pre-experiment covariates and outcome metrics, greatly limiting the effectiveness of variance reduction. In this paper, we develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics and construct a robust average treatment effect estimator in online controlled experiments, which we call STATE. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers and achieves significant variance reduction. Moreover, we extend the STATE method from count metrics to ratio metrics by utilizing linear transformation that preserves unbiased estimation, whose variance reduction is more complex but less investigated in existing works. Finally, both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method. Compared with the state-of-the-art estimators (CUPAC/MLRATE), STATE achieves over 50% variance reduction, indicating it can reach the same statistical power with only half of the observations, or half the experimental duration.

Figures (4)

• Figure 1: The distribution of residuals for user GMV on the Meituan food delivery platform. The Y-axis represents the sample count. The X-axis represents the residuals between the real value and the model predicted value, where the extremely large values are clipped to the upper bound.
• Figure 2: Simulation results of ATE estimation
• Figure 3: Impact of proportions of outliers
• Figure 4: Empirical results of ATE estimation