Large-Scale Metric Computation in Online Controlled Experiment Platform

TL;DR

This work shows how metric computation in WeChat experiment platform can be done efficiently using bit-sliced index (BSI) arithmetic, and results are presented, showing that the BSI arithmetic approach is very suitable for large-scale metric computation scenarios.

Abstract

Online controlled experiment (also called A/B test or experiment) is the most important tool for decision-making at a wide range of data-driven companies like Microsoft, Google, Meta, etc. Metric computation is the core procedure for reaching a conclusion during an experiment. With the growth of experiments and metrics in an experiment platform, computing metrics efficiently at scale becomes a non-trivial challenge. This work shows how metric computation in WeChat experiment platform can be done efficiently using bit-sliced index (BSI) arithmetic. This approach has been implemented in a real world system and the performance results are presented, showing that the BSI arithmetic approach is very suitable for large-scale metric computation scenarios.

Figures (9)

• Figure 1: BSI Example
• Figure 2: BSI Addition
• Figure 3: Position Encoding Example of Segment 0
• Figure 4: Value Range Cardinalities Distribution of 5890 Real World Metrics in One Day
• Figure 5: Metric Value Distribution Examples