Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Normality: Reliable A/B Testing with Non-Gaussian Data

Junpeng Gong, Chunkai Wang, Hao Li, Jinyong Ma, Haoxuan Li, Xu He

TL;DR

The paper tackles unreliable Type I error control in online A/B testing caused by non-Gaussian data and unequal group allocations. It derives explicit minimum sample-size thresholds N_min^(1) and N_min^(2) that account for skewness and kurtosis through a unified framework, enabling reliable inference in general online settings. For small samples, it introduces an Edgeworth-based p-value correction using plug-in estimates of higher moments, improving balanced error control across both tails. Empirical results on synthetic and real platform data show substantial improvements in reliability and potential cost savings, highlighting practical impact for large-scale online experiments.

Abstract

A/B testing has become the cornerstone of decision-making in online markets, guiding how platforms launch new features, optimize pricing strategies, and improve user experience. In practice, we typically employ the pairwise $t$-test to compare outcomes between the treatment and control groups, thereby assessing the effectiveness of a given strategy. To be trustworthy, these experiments must keep Type I error (i.e., false positive rate) under control; otherwise, we may launch harmful strategies. However, in real-world applications, we find that A/B testing often fails to deliver reliable results. When the data distribution departs from normality or when the treatment and control groups differ in sample size, the commonly used pairwise $t$-test is no longer trustworthy. In this paper, we quantify how skewed, long tailed data and unequal allocation distort error rates and derive explicit formulas for the minimum sample size required for the $t$-test to remain valid. We find that many online feedback metrics require hundreds of millions samples to ensure reliable A/B testing. Thus we introduce an Edgeworth-based correction that provides more accurate $p$-values when the available sample size is limited. Offline experiments on a leading A/B testing platform corroborate the practical value of our theoretical minimum sample size thresholds and demonstrate that the corrected method substantially improves the reliability of A/B testing in real-world conditions.

Beyond Normality: Reliable A/B Testing with Non-Gaussian Data

TL;DR

The paper tackles unreliable Type I error control in online A/B testing caused by non-Gaussian data and unequal group allocations. It derives explicit minimum sample-size thresholds N_min^(1) and N_min^(2) that account for skewness and kurtosis through a unified framework, enabling reliable inference in general online settings. For small samples, it introduces an Edgeworth-based p-value correction using plug-in estimates of higher moments, improving balanced error control across both tails. Empirical results on synthetic and real platform data show substantial improvements in reliability and potential cost savings, highlighting practical impact for large-scale online experiments.

Abstract

A/B testing has become the cornerstone of decision-making in online markets, guiding how platforms launch new features, optimize pricing strategies, and improve user experience. In practice, we typically employ the pairwise -test to compare outcomes between the treatment and control groups, thereby assessing the effectiveness of a given strategy. To be trustworthy, these experiments must keep Type I error (i.e., false positive rate) under control; otherwise, we may launch harmful strategies. However, in real-world applications, we find that A/B testing often fails to deliver reliable results. When the data distribution departs from normality or when the treatment and control groups differ in sample size, the commonly used pairwise -test is no longer trustworthy. In this paper, we quantify how skewed, long tailed data and unequal allocation distort error rates and derive explicit formulas for the minimum sample size required for the -test to remain valid. We find that many online feedback metrics require hundreds of millions samples to ensure reliable A/B testing. Thus we introduce an Edgeworth-based correction that provides more accurate -values when the available sample size is limited. Offline experiments on a leading A/B testing platform corroborate the practical value of our theoretical minimum sample size thresholds and demonstrate that the corrected method substantially improves the reliability of A/B testing in real-world conditions.

Paper Structure

This paper contains 18 sections, 3 theorems, 28 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Non-Normality of Test Statistics in Online Controlled Experiments
  4. Reliable Testing for Online Controlled Experiments
  5. Preliminaries
  6. Preliminaries
  7. Two-Sample t-Tests in A/B testing
  8. Error Metric and Problem Statement
  9. Methodology
  10. Finite-Sample Error in A/B Testing
  11. Minimum Sample Size for Reliable $t$-test
  12. Edgeworth-Based Correction for Small Samples
  13. Experiment
  14. Synthetic Data Experiment
  15. Real Data Experiment
  16. ...and 3 more sections

Key Result

Theorem 1

Suppose $X_1,\ldots,X_{n_x}$ are independent and identically distributed with $E|X_1|^4<\infty$, and $Y_1,\ldots,Y_{n_y}$ are independent and identically distributed with $E|Y_1|^4<\infty$. Assume that the two samples ${X_i}$ and ${Y_j}$ are independent. Then, for the mean difference $D = \bar{Y} -

Figures (3)

  • Figure 1: Empirical density of the Welch's $t$-test statistic $T$ (blue), based on real data on the number of videos published by users of a popular online platform, with significance level $\alpha = 0.05$. The control and treatment sample sizes are 100 and 1,000, and the empirical distribution is compared with the standard normal $N(0,1)$ density (orange).
  • Figure 2: Empirical vs. theoretical minimal sample sizes $N_{\min}^{(1)}$ and $N_{\min}^{(2)}$ under $\alpha=0.1$ across varying tolerance levels $\epsilon$ for (left) publish count and (right) live duration datasets.
  • Figure 3: In-depth comparison at $\alpha=0.10$ and $\epsilon=0.03$ for two metrics (publish count, live duration) under two statistics (Welch $t$ vs. Edgeworth-corrected $p_c$).

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3