Bridging Control Variates and Regression Adjustment in A/B Testing: From Design-Based to Model-Based Frameworks

TL;DR

This paper tackles variance reduction in online A/B testing by formally linking control variates with regression adjustment and extending the analysis from design-based to model-based frameworks. It introduces multiple estimators for the control variate coefficient and corresponding ATE estimators, then establishes that control variates with group-specific coefficients $\widehat{\delta}_3$ are asymptotically equivalent to regression adjustment with interaction terms, and often provide superior precision, especially under heterogeneous treatment effects. The work shows that, in the design-based setting, $\widehat{\delta}_3$ is at least as efficient as other estimators, while in the model-based setting the uncorrected variance for $\widehat{\delta}_3$ can be aggressive and requires a correction to match $\widehat{\delta}_2$’s performance; a variance-correction is proposed and shown to preserve or improve power. The theoretical results are validated through extensive simulations and real ByteDance data, and the recommended approach—control variates with group-specific $\theta$ estimates—has been implemented on ByteDance’s platform, improving testing sensitivity while maintaining valid inference.

Abstract

A B testing serves as the gold standard for large scale, data driven decision making in online businesses. To mitigate metric variability and enhance testing sensitivity, control variates and regression adjustment have emerged as prominent variance reduction techniques, leveraging pre experiment data to improve estimator performance. Over the past decade, these methods have spawned numerous derivatives, yet their theoretical connections and comparative properties remain underexplored. In this paper, we conduct a comprehensive analysis of their statistical properties, establish a formal bridge between the two frameworks in practical implementations, and extend the investigation from design based to model-based frameworks. Through simulation studies and real world experiments at ByteDance, we validate our theoretical insights across both frameworks. Our work aims to provide rigorous guidance for practitioners in online controlled experiments, addressing critical considerations of internal and external validity. The recommended method control variates with group specific coefficient estimates has been fully implemented and deployed on ByteDance's experimental platform.

Figures (5)

• Figure 1: Asymptotically discrepancies between $\widehat{\delta}_3$ and other estimators, scaled as $\sqrt n\cdot(\widehat{\text{ATE}} - \widehat{\delta}_3)$.
• Figure 2: Box plots of the variance estimates of ATE estimators, scaled by $n \times \widehat{\text{var}}$.
• Figure 3: Power - ATE/HTE curves for estimators under varying group assignment probabilities.
• Figure 4: Asymptotically discrepancies between $\widehat{\delta}_3$ and other estimators, scaled as $\sqrt n\cdot(\widehat{\text{ATE}} - \widehat{\delta}_3)$.
• Figure 5: Box plots of the variance estimates of ATE estimators, scaled by $n \times \widehat{\text{var}}$.

Theorems & Definitions (10)

• Theorem 2.1
• Theorem 2.2
• Theorem 3.1
• Theorem 4.1
• Theorem 4.2
• Theorem 5.1
• Theorem 5.2
• Theorem 5.3
• Lemma A.1: Finite-population version of the Weak Law of Large Numbers
• Lemma A.2