Dynamic Prior Thompson Sampling for Cold-Start Exploration in Recommender Systems

TL;DR

Cold-start exploration in large-scale recommender systems is hindered by batched updates, where new items can accrue exposure with minimal data for hours, and uniform Beta(1,1) priors overestimate unseen-item success. Dynamic Prior Thompson Sampling replaces the fixed prior with a data-driven prior mean $q_j$ for each new arm, chosen to enforce $P(X_j > Y_k)=\varepsilon$ at introduction using a closed-form quadratic solution derived under a normal approximation to Beta posteriors, yielding predictable, tunable exploration. The paper provides a complete derivation, Monte Carlo validation showing $P(X_j > Y_k) ≈ \varepsilon$ within $0.01$ across configurations, and extensive simulation and online experiments demonstrating up to 9.5% gains in cumulative reward and reductions in regretted impressions in production. The approach aligns exploration intensity with the current production baseline, remains intrinsic to Thompson Sampling, and is practical under latency and batching, offering a path to safer, more efficient cold-start handling with potential extensions to contextual and multi-objective settings.

Abstract

Cold-start exploration is a core challenge in large-scale recommender systems: new or data-sparse items must receive traffic to estimate value, but over-exploration harms users and wastes impressions. In practice, Thompson Sampling (TS) is often initialized with a uniform Beta(1,1) prior, implicitly assuming a 50% success rate for unseen items. When true base rates are far lower, this optimistic prior systematically over-allocates to weak items. The impact is amplified by batched policy updates and pipeline latency: for hours, newly launched items can remain effectively "no data," so the prior dominates allocation before feedback is incorporated. We propose Dynamic Prior Thompson Sampling, a prior design that directly controls the probability that a new arm outcompetes the incumbent winner. Our key contribution is a closed-form quadratic solution for the prior mean that enforces P(X_j > Y_k) = epsilon at introduction time, making exploration intensity predictable and tunable while preserving TS Bayesian updates. Across Monte Carlo validation, offline batched simulations, and a large-scale online experiment on a thumbnail personalization system serving millions of users, dynamic priors deliver precise exploration control and improved efficiency versus a uniform-prior baseline.

Figures (9)

• Figure 1: (Theoretical validation; Monte Carlo) Empirical exploration probability versus target $\varepsilon$ across 800 configurations. Each subplot corresponds to a target probability $\varepsilon$ and prior strength $r$, with colors indicating incumbent success rates $p$. The red dashed lines mark targets and error bars show Monte Carlo standard errors. The close alignment demonstrates that the closed-form dynamic prior achieves the intended exploration probability over a wide range of operational regimes.
• Figure 2: (Simulation study) Cumulative reward trajectories for all methods and configurations. The vertical dashed line at batch 5 indicates the introduction of the new arm. Dynamic Prior configurations consistently outperform baselines, with error bars showing 95% confidence intervals for final rewards.
• Figure 3: (Simulation study) Allocation patterns for Dynamic Prior across parameter combinations. Each subplot shows the per-batch traffic distribution among the five initial arms and the new arm over 10 batches. Allocation adapts smoothly as evidence accumulates, reducing traffic to the weak new arm while preserving calibrated exploration.
• Figure 4: (Simulation study) Uniform-prior allocation pattern showing persistent over-allocation to the weak new arm due to optimistic $\mathrm{Beta}(1,1)$ assumptions. This illustrates the mismatch between uniform priors and low-success-rate operational environments that motivates dynamic priors.
• Figure 5: (Simulation study) Allocation patterns for fixed-horizon forced exploration across parameter combinations. Yellow regions indicate forced exploration periods where the new arm receives guaranteed traffic. The resulting sharp phase transitions contrast with the smooth, posterior-driven adaptation of Dynamic Prior.