Variance Reduction in Ratio Metrics for Efficient Online Experiments

TL;DR

The paper addresses the inefficiency of online A/B tests for ratio metrics by applying variance-reduction techniques to improve test sensitivity. It formalizes z-testing for ratio metrics, employing the Delta method and linearisation to handle variance, and proposes a CUPED-inspired framework augmented with gradient-boosted decision tree predictors as control variates. Empirical results on ShareChat show that GBDT-based covariates significantly boost sensitivity in 77% of cases, while relying solely on pre-experiment covariates can hurt accuracy; combining covariates offers limited gains due to bias, yielding a median relative $z$-score of $1.19$ and enabling roughly $30\%$ fewer data points per experiment. The findings advocate for unbiased estimators in variance reduction to achieve faster, cost-effective A/B testing, with practical guidance for scalable deployment and future work on variance-bias trade-offs in ratio-metric settings.

Abstract

Online controlled experiments, such as A/B-tests, are commonly used by modern tech companies to enable continuous system improvements. Despite their paramount importance, A/B-tests are expensive: by their very definition, a percentage of traffic is assigned an inferior system variant. To ensure statistical significance on top-level metrics, online experiments typically run for several weeks. Even then, a considerable amount of experiments will lead to inconclusive results (i.e. false negatives, or type-II error). The main culprit for this inefficiency is the variance of the online metrics. Variance reduction techniques have been proposed in the literature, but their direct applicability to commonly used ratio metrics (e.g. click-through rate or user retention) is limited. In this work, we successfully apply variance reduction techniques to ratio metrics on a large-scale short-video platform: ShareChat. Our empirical results show that we can either improve A/B-test confidence in 77% of cases, or can retain the same level of confidence with 30% fewer data points. Importantly, we show that the common approach of including as many covariates as possible in regression is counter-productive, highlighting that control variates based on Gradient-Boosted Decision Tree predictors are most effective. We discuss the practicalities of implementing these methods at scale and showcase the cost reduction they beget.