Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved prediction of future user activity in online A/B testing

Lorenzo Masoero, Mario Beraha, Thomas Richardson, Stefano Favaro

TL;DR

This work tackles the problem of predicting future user activity in online A/B tests, including both the number of new users and their re-trigger rates. It introduces a Bayesian nonparametric model built on a negative-binomial likelihood and a negative-binomial scaled-stable process prior (NBP with SBSP), enabling exact posterior and predictive forms without MCMC. The authors derive closed-form expressions for key quantities, provide an urn-based sampling scheme for efficient simulation, and propose two hyperparameter-estimation strategies (maximum marginal likelihood and regression-based). Empirical results on synthetic and real data show the BNP predictor is competitive with, and often superior to, existing approaches for predicting future activity and total retrigger rates, with clear implications for experiment duration planning and cost reduction in online A/B testing. The work also demonstrates the practical viability of BNP methods in large-scale online experiments and outlines promising avenues for extending predictions to more granular re-trigger dynamics and long-term effects.

Abstract

In online randomized experiments or A/B tests, accurate predictions of participant inclusion rates are of paramount importance. These predictions not only guide experimenters in optimizing the experiment's duration but also enhance the precision of treatment effect estimates. In this paper we present a novel, straightforward, and scalable Bayesian nonparametric approach for predicting the rate at which individuals will be exposed to interventions within the realm of online A/B testing. Our approach stands out by offering dual prediction capabilities: it forecasts both the quantity of new customers expected in future time windows and, unlike available alternative methods, the number of times they will be observed. We derive closed-form expressions for the posterior distributions of the quantities needed to form predictions about future user activity, thereby bypassing the need for numerical algorithms such as Markov chain Monte Carlo. After a comprehensive exposition of our model, we test its performance on experiments on real and simulated data, where we show its superior performance with respect to existing alternatives in the literature.

Improved prediction of future user activity in online A/B testing

TL;DR

This work tackles the problem of predicting future user activity in online A/B tests, including both the number of new users and their re-trigger rates. It introduces a Bayesian nonparametric model built on a negative-binomial likelihood and a negative-binomial scaled-stable process prior (NBP with SBSP), enabling exact posterior and predictive forms without MCMC. The authors derive closed-form expressions for key quantities, provide an urn-based sampling scheme for efficient simulation, and propose two hyperparameter-estimation strategies (maximum marginal likelihood and regression-based). Empirical results on synthetic and real data show the BNP predictor is competitive with, and often superior to, existing approaches for predicting future activity and total retrigger rates, with clear implications for experiment duration planning and cost reduction in online A/B testing. The work also demonstrates the practical viability of BNP methods in large-scale online experiments and outlines promising avenues for extending predictions to more granular re-trigger dynamics and long-term effects.

Abstract

In online randomized experiments or A/B tests, accurate predictions of participant inclusion rates are of paramount importance. These predictions not only guide experimenters in optimizing the experiment's duration but also enhance the precision of treatment effect estimates. In this paper we present a novel, straightforward, and scalable Bayesian nonparametric approach for predicting the rate at which individuals will be exposed to interventions within the realm of online A/B testing. Our approach stands out by offering dual prediction capabilities: it forecasts both the quantity of new customers expected in future time windows and, unlike available alternative methods, the number of times they will be observed. We derive closed-form expressions for the posterior distributions of the quantities needed to form predictions about future user activity, thereby bypassing the need for numerical algorithms such as Markov chain Monte Carlo. After a comprehensive exposition of our model, we test its performance on experiments on real and simulated data, where we show its superior performance with respect to existing alternatives in the literature.
Paper Structure (36 sections, 8 theorems, 69 equations, 15 figures)

This paper contains 36 sections, 8 theorems, 69 equations, 15 figures.

Table of Contents

  1. Introduction
  2. A nonparametric Bayesian model to predict future activity in online A/B tests
  3. The negative-binomial stable-beta scaled-process
  4. Posterior and predictive characterization of the negative binomial scaled-stable process
  5. Predicting future user re-trigger rates with the negative binomial scaled stable process
  6. Marginal representation and sampling scheme
  7. Inference
  8. Maximum marginal likelihood
  9. Regression-based approach
  10. Experimental Results
  11. Competing methods and evaluation framework
  12. Synthetic data
  13. Data from the model
  14. Synthetic data from Zipf distributions
  15. Experiments on real data
  16. ...and 21 more sections

Key Result

Proposition 1

Let $f:=f_{\Delta_{0, h}\mid Z_{1:D_{0}}}$ denote the posterior density of the largest jump. Under the model of eq:model and given $Z_{1:D_{0}}$, it holds That is, $\Delta_{0, h}^{-\sigma}\mid Z_{1:D_{0}} \sim \mathrm{Gamma}(N_{D_{0}}+c+1, \beta + \psi_0^{(D_{0})})$. Here, letting $B(a,b)$ be the beta function with parameters $a$, $b$, we use the notation

Figures (15)

  • Figure 1: Sketch of an online A/B tests. Three units first trigger and subsequently re-trigger in the experiment.
  • Figure 2: Prediction of the arrival of new users (top left, $U_{0}^{(d)}$, other subplots, $U_{D_{0}}^{(d, j)}$ for $j = 1, \ldots, 5$). We plot the value of the statistic (vertical axis) as the number of days $d$ increases (horizontal axis). We compare the true value in the unobservable sample (solid blue line) to the predicted posterior mean with a centred 95% credible interval using the fitted hyperparameters (red, dotted) and using the true hyperparameters (green, dash-dot). Shaded blue lines report additional $M=100$ draws from the model.
  • Figure 3: Prediction of future user activity (left, $S_{D_{0}}^{(d)}$, center, $\sum_j U_{D_{0}}^{(d, j)}$, right $T_{D_{0}}^{(d)}$). We plot the value of the statistic (vertical axis) as the number of days $d$ increases (horizontal axis). We compare the true value in future samples (solid blue line) to the predicted posterior mean using the fitted (red) and the true (green) hyperparameters.
  • Figure 4: Prediction accuracy $v_{D_{0}}^{(D_{1})}$ of the Bayesian nonparametric predictor $\hat{U}_{D_{0}}^{(D_{1})}$ on data from the model.
  • Figure 5: Prediction accuracy $v_{D_{0}}^{(D_{1})}$ of predictors $\hat{U}_{D_{0}}^{(D_{1})}$ on synthetic data from the Zipfian model for different choices of the parameter $\tau$. Here, $D_{0}=10$ and $D_{1}=50$.
  • ...and 10 more figures

Theorems & Definitions (15)

  • Remark 1: Model specification
  • Proposition 1: Distribution of the largest jump
  • Proposition 2: Posterior representation
  • Proposition 3: Predictive representation
  • Proposition 4: Marginal distribution of the observed sample
  • Proposition 5: Number of new users
  • Proposition 6: Number of new users with frequency
  • Corollary 7: Future re-trigger rates, old users
  • Corollary 8: Total future re-trigger rates
  • proof
  • ...and 5 more