A Bayesian Model for Online Activity Sample Sizes

TL;DR

We address the challenge of predicting the number of new online participants in a future period given early-period data, explicitly accounting for heterogeneity in initiation times. The authors propose a hierarchical Bayesian Beta-Geometric model with censoring, enabling posterior predictive sampling of future accrual without reliance on heavy MCMC. They derive posteriors for per-user probabilities $\pi_i$ and hyperparameters $\alpha$, $\beta$, and show how to forecast second-period arrivals even when the total population size is unknown, using a robust plug-in approach for $n_0$ when necessary. Empirical evaluation on a large Amazon dataset demonstrates competitive predictive performance against baselines (e.g., ARIMA, log-linear, Random Forest) and supports practical deployment, with explicit discussion of robustness to $n_0$ and benefits of heterogeneity modeling.

Abstract

In many contexts it is useful to predict the number of individuals in some population who will initiate a particular activity during a given period. For example, the number of users who will install a software update, the number of customers who will use a new feature on a website or who will participate in an A/B test. In practical settings, there is heterogeneity amongst individuals with regard to the distribution of time until they will initiate. For these reasons it is inappropriate to assume that the number of new individuals observed on successive days will be identically distributed. Given observations on the number of unique users participating in an initial period, we present a simple but novel Bayesian method for predicting the number of additional individuals who will participate during a subsequent period. We illustrate the performance of the method in predicting sample size in online experimentation.

Figures (4)

• Figure 1: Graphical Model: The hierarchical Beta-Geometric model. $(\alpha,\beta)$ are hyper-parameters; for an individual $i$, $\pi_i$ is the individual-specific probability of first participating on a given day, having not participated previously; $X_i$ indicates either the day on which the customer $i$ first participates in the first period or whether the individual did not participate in the first period ($X_i=0$). Shaded nodes are observed; nodes shaded red indicate individuals who didn't participate in the first period. $X^*_i$ is a prediction of when an individual who did not participate in the first period will first participate in the second period, or if they will again not participate ($X_i^*=0$).
• Figure 2: Illustrative Example: For the sake of simplicity, we label customers participating in the experiment first time as "new customers". (a) the X-axis represents the number of days since the start of the experiment; the Y-axis represents the number of new customers who first participate in the experiment during each day in the initial period. (b) Scatter plot of 1000 pairs of ($\alpha$, $\beta$) values sampled from Equation (\ref{['eq:posterior-over-hyperparams']}) given the data in Figure \ref{['fig:data']} with $\lambda=10$. Contour lines show the density of ($\alpha$, $\beta$) and nested contours indicate regions of higher local density. (c) The Y-axis represents the predicted number of new customers participating in the experiment at week $k$, where $k \in\{1, 2, 3, 4\}$. The plot displays 1000 posterior draws for the prediction functions corresponding to the ($\alpha$, $\beta$) samples in Figure \ref{['fig:data']}.
• Figure 3: Results For Illustrative Example: (a) Box plots of the predicted number of new customers participating in the experiment in Figure \ref{['fig:data']} at week 2 using different lambda values, $\lambda\in \{1, 2, \ldots ,30\}$. The X-axis is the value of lambda. Left bottom corner of the plot is $(0, 0)$. Each boxplot displays the distribution of 1000 draws from the posterior predictive distribution for the number of new customers at week 2 obtained when using a specific value of lambda. The Red dotted line is the ground truth observed at week 2. (b) Comparison of the predicted number of new customers participating in the experiment shown in Figure \ref{['fig:data']} at the $k$-th week using different methods. The X-axis represents the week index and the Y-axis represents the number of customers first participating in the experiment at week $k$. The true number for week 1 is observed; the numbers for weeks 2, 3, 4, 5 are predicted using different methods. The red curve shows the predictions using the proposed Bayesian model. Predictions from the Log Linear and Random Forest models are given by the blue and purple curves respectively. The ground truth is given by the green curve.
• Figure 4: Meta-Analysis Results: The vertical axes show the true number of new customers participating in an experiment at the $k$-th week; the horizontal axes give the predicted count of new customers participating based on the first week of data. The $45^\circ$ line, corresponding to perfect prediction, is shown in red. Scatter plots contain predictions at week $k$ for experiments running for at least $k$ weeks. The left panel shows results for $981$ experiments that ran for $2$ weeks; the right panel shows $488$ experiments that ran for $4$ weeks. The plots show the performance of the proposed predictor (top), random forest regression (middle) and the Log Linear predictor (bottom) at (a) 2 weeks and (b) 4 weeks. For visualization purposes, outliers in the Log Linear predictions are Winsorized at the maximum value on the horizontal axis and indicated by $+$.