Learning Metrics that Maximise Power for Accelerated A/B-Tests

TL;DR

The paper tackles the inefficiency of relying on delayed North Star metrics in online experimentation by learning short-term signals that maximise statistical power toward the North Star. It introduces a linear metric transformation $\omega = \boldsymbol{\mu} w^{\intercal}$ and optimises the weight vector $w$ to improve $z$-scores or $\log p$-values, with corrections for multiple testing and sequential peeking. A gradient-based objective combines known outcomes, inconclusive outcomes, and A/A controls, and a spherical regularisation term speeds up convergence for scale-free objectives without changing optima. Empirical results on large-scale ShareChat and Moj datasets show that learnt metrics can boost statistical power by up to $210\%$ and reduce required sample sizes to as low as $12\%$ of the North Star’s, enabling faster, cheaper, and more reliable A/B testing. The work also provides practical guidance on avoiding ratio-metric pitfalls and highlights the value of per-user aggregations, with the learnt metrics already deployed in production. $\,$

Abstract

Online controlled experiments are a crucial tool to allow for confident decision-making in technology companies. A North Star metric is defined (such as long-term revenue or user retention), and system variants that statistically significantly improve on this metric in an A/B-test can be considered superior. North Star metrics are typically delayed and insensitive. As a result, the cost of experimentation is high: experiments need to run for a long time, and even then, type-II errors (i.e. false negatives) are prevalent. We propose to tackle this by learning metrics from short-term signals that directly maximise the statistical power they harness with respect to the North Star. We show that existing approaches are prone to overfitting, in that higher average metric sensitivity does not imply improved type-II errors, and propose to instead minimise the $p$-values a metric would have produced on a log of past experiments. We collect such datasets from two social media applications with over 160 million Monthly Active Users each, totalling over 153 A/B-pairs. Empirical results show that we are able to increase statistical power by up to 78% when using our learnt metrics stand-alone, and by up to 210% when used in tandem with the North Star. Alternatively, we can obtain constant statistical power at a sample size that is down to 12% of what the North Star requires, significantly reducing the cost of experimentation.

Figures (7)

• Figure 1: Visualising our proposed optimisation objectives for learnt metrics, as a function of their $z$-score.
• Figure 2: Directly maximising $z$-scores yields a scale-free objective that is not amenable to efficient optimisation with gradient ascent (left). Spherical regularisation retains all optima, whilst providing a gradient direction that is more aligned (right).
• Figure 3: Learnt metrics and their agreement with known North Star outcomes over varying significance levels. A learnt metric that maximises the average $z$-score exhibits significant type-III/S error, which can be alleviated by minimising $p$-values instead.
• Figure 4: When considering only learnt metrics, we improve type-II error significantly without hurting specificity. At $\alpha=0.05$, we increase statistical power by 67% (upper plot). In conjunction with the North Star and top proxy metrics, Bonferroni corrections are slightly conservative (type-I error $< \alpha$), and allow us to improve statistical power by 133% for $\alpha=0.05$ (lower plot).
• Figure 5: When considering learnt metrics in conjunction with the North Star and top proxy metrics, we require significantly reduced sample sizes to obtain the same statistical significance level as we would get from the North Star.