Algorithms for Adaptive Experiments that Trade-off Statistical Analysis with Reward: Combining Uniform Random Assignment and Reward Maximization

TL;DR

This work tackles the challenge of balancing reward maximization with reliable statistical inference in adaptive two-arm experiments. It introduces TS-PostDiff, a hybrid algorithm that blends Thompson Sampling with Uniform Random allocation, where the UR probability is driven by the posterior likelihood that the arm difference is small, controlled by a threshold c. Through simulations and real-world-inspired studies, TS-PostDiff demonstrates improved trade-offs: near-UR performance when differences are small to control false positives and increase power, and TS-like reward optimization when differences are large. The paper also compares TT-TS and TS-ProbClip, analyzes the impact of the small-difference parameter c, and provides practical guidelines for deploying statisti cally sensitive adaptive designs with clear limitations and avenues for future work.

Abstract

Traditional randomized A/B experiments assign arms with uniform random (UR) probability, such as 50/50 assignment to two versions of a website to discover whether one version engages users more. To more quickly and automatically use data to benefit users, multi-armed bandit algorithms such as Thompson Sampling (TS) have been advocated. While TS is interpretable and incorporates the randomization key to statistical inference, it can cause biased estimates and increase false positives and false negatives in detecting differences in arm means. We introduce a more Statistically Sensitive algorithm, TS-PostDiff (Posterior Probability of Small Difference), that mixes TS with traditional UR by using an additional adaptive step, where the probability of using UR (vs TS) is proportional to the posterior probability that the difference in arms is small. This allows an experimenter to define what counts as a small difference, below which a traditional UR experiment can obtain informative data for statistical inference at low cost, and above which using more TS to maximize user benefits is key. We evaluate TS-PostDiff against UR, TS, and two other TS variants designed to improve statistical inference. We consider results for the common two-armed experiment across a range of settings inspired by real-world applications. Our results provide insight into when and why TS-PostDiff or alternative approaches provide better tradeoffs between benefiting users (reward) and statistical inference (false positive rate and power). TS-PostDiff's adaptivity helps efficiently reduce false positives and increase statistical power when differences are small, while increasing reward more when differences are large. The work highlights important considerations for future Statistically Sensitive algorithm development that balances reward and statistical analysis in adaptive experimentation.

Figures (5)

• Figure 1: Plots showing the evolution of Reward, Power, and Prop Opt./Sup. across differnt arm differences. The sample size is fixed at 785. The algorithm parameters for TS-PostDiff, TT-TS and TS-ProbClip are chosen to align their FPR at 0.06 when there's no arm difference. The red line (at arm difference = 0.175) indicates the location where the Reward from TS-PostDiff catches up with the other two mixture algorithms' Reward performance. In the Power Gain plot, we plot FPR instead when there's no arm difference, which explains why TS comes to the top.
• Figure 2: Power-reward plots for Uniform Random, TS, TS-PostDiff, $\epsilon$/TT-TS and TS-ProbClip under sample size 197 and 785. The blue dashed lines separate simulation results for the different arm differences/arm differences.
• Figure 3: Values of $\hat{\phi_t}$, the estimated probability of choosing actions uniformly, for different sample sizes and values of the 'small-difference' parameter $c$ across $10,000$ simulations for the arm difference $0.1$ (right) and $0$ (left). The x-axis denotes sample size and the y-axis shows $\hat{\phi}$. We approximate $\phi$, the true probability of choosing actions uniformly randomly, as $\hat{\phi}$ by taking the proportion of times $|p_1 - p_2| < c$ across $10,000$ simulations, for the given sample size shown on the x-axis. For values of $c$ greater than the arm difference, $\hat{\phi}$ converges to $1$, meaning TS-PostDiff allocates actions with Uniform Random. For values of $c$ less than the arm difference, $\hat{\phi}$ converges to $0$, meaning TS-PostDiff behaves like standard TS.
• Figure 4: As $c$ increases, FPR (left) decreases and Power (middle) increases, while Reward (right) decreases. Improvements to Power and FPR diminish as $c$ increases, while the impact on Reward is roughly linear in $c$. Figures show results for a sample size of $785$ simulated participants.
• Figure 5: Real world deployments where we tested the behavior of TS-PostDiff while comparing it with Uniform Random allocation. Case 1 (Top): 473 participants were randomly allocated to the arms, showing an arm difference of $0.016$; the other 438 students were allocated by a TS-PostDiff policy with $c = 0.1$, getting an arm difference of $0.27$, showing that TS-PostDiff chose to explore given that the estimated arm difference was small compared with the parameter $c$. Case 2 (Bottom): 584 participants were randomly allocated to the arms, showing an arm difference of $0.045$; the other 587 students were allocated by a TS-PostDiff policy with $c = 0.02$, getting an arm difference of $0.062$, showing that TS-PostDiff chose to exploit the arm with better mean given that the estimated arm difference was large compared with the parameter $c$.