Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thompson sampling: Precise arm-pull dynamics and adaptive inference

Qiyang Han

TL;DR

This work provides a complete characterization of the arm-pull dynamics under generalized Gaussian Thompson sampling, revealing a sharp dichotomy: suboptimal arms or a unique optimal arm yield asymptotically deterministic pulls, while multiple optimal arms induce nontrivial stochastic limits described by invariant measures of time-changed SDEs. The authors show that normalized empirical means converge to Gaussian limits for stable arms and to a non-Gaussian, semi-universal limit for unstable arms, enabling confidence intervals even in non-normal regimes. A duality between arm stability and interaction with statistical noise unifies stability phenomena across bandit algorithms and informs adaptive inference strategies. The paper introduces two novel proof techniques—the inverse-process approach for suboptimal arms and a time-change renormalization for optimal arms—paired with advanced tools from stochastic analysis (parabolic Hörmander condition and Stroock-Varadhan support theorem) to establish existence and uniqueness of invariant measures and to derive distributional limits that underpin practical inference in adaptive data settings.

Abstract

Adaptive sampling schemes are well known to create complex dependence that may invalidate conventional inference methods. A recent line of work shows that this need not be the case for UCB-type algorithms in multi-armed bandits. A central emerging theme is a `stability' property with asymptotically deterministic arm-pull counts in these algorithms, making inference as easy as in the i.i.d. setting. In this paper, we study the precise arm-pull dynamics in another canonical class of Thompson-sampling type algorithms. We show that the phenomenology is qualitatively different: the arm-pull count is asymptotically deterministic if and only if the arm is suboptimal or is the unique optimal arm; otherwise it converges in distribution to the unique invariant law of an SDE. This dichotomy uncovers a unifying principle behind many existing (in)stability results: an arm is stable if and only if its interaction with statistical noise is asymptotically negligible. As an application, we show that normalized arm means obey the same dichotomy, with Gaussian limits for stable arms and a semi-universal, non-Gaussian limit for unstable arms. This not only enables the construction of confidence intervals for the unknown mean rewards despite non-normality, but also reveals the potential of developing tractable inference procedures beyond the stable regime. The proofs rely on two new approaches. For suboptimal arms, we develop an `inverse process' approach that characterizes the inverse of the arm-pull count process via a Stieltjes integral. For optimal arms, we adopt a reparametrization of the arm-pull and noise processes that reduces the singularity in the natural SDE to proving the uniqueness of the invariant law of another SDE. We prove the latter by a set of analytic tools, including the parabolic Hörmander condition and the Stroock-Varadhan support theorem.

Thompson sampling: Precise arm-pull dynamics and adaptive inference

TL;DR

This work provides a complete characterization of the arm-pull dynamics under generalized Gaussian Thompson sampling, revealing a sharp dichotomy: suboptimal arms or a unique optimal arm yield asymptotically deterministic pulls, while multiple optimal arms induce nontrivial stochastic limits described by invariant measures of time-changed SDEs. The authors show that normalized empirical means converge to Gaussian limits for stable arms and to a non-Gaussian, semi-universal limit for unstable arms, enabling confidence intervals even in non-normal regimes. A duality between arm stability and interaction with statistical noise unifies stability phenomena across bandit algorithms and informs adaptive inference strategies. The paper introduces two novel proof techniques—the inverse-process approach for suboptimal arms and a time-change renormalization for optimal arms—paired with advanced tools from stochastic analysis (parabolic Hörmander condition and Stroock-Varadhan support theorem) to establish existence and uniqueness of invariant measures and to derive distributional limits that underpin practical inference in adaptive data settings.

Abstract

Adaptive sampling schemes are well known to create complex dependence that may invalidate conventional inference methods. A recent line of work shows that this need not be the case for UCB-type algorithms in multi-armed bandits. A central emerging theme is a `stability' property with asymptotically deterministic arm-pull counts in these algorithms, making inference as easy as in the i.i.d. setting. In this paper, we study the precise arm-pull dynamics in another canonical class of Thompson-sampling type algorithms. We show that the phenomenology is qualitatively different: the arm-pull count is asymptotically deterministic if and only if the arm is suboptimal or is the unique optimal arm; otherwise it converges in distribution to the unique invariant law of an SDE. This dichotomy uncovers a unifying principle behind many existing (in)stability results: an arm is stable if and only if its interaction with statistical noise is asymptotically negligible. As an application, we show that normalized arm means obey the same dichotomy, with Gaussian limits for stable arms and a semi-universal, non-Gaussian limit for unstable arms. This not only enables the construction of confidence intervals for the unknown mean rewards despite non-normality, but also reveals the potential of developing tractable inference procedures beyond the stable regime. The proofs rely on two new approaches. For suboptimal arms, we develop an `inverse process' approach that characterizes the inverse of the arm-pull count process via a Stieltjes integral. For optimal arms, we adopt a reparametrization of the arm-pull and noise processes that reduces the singularity in the natural SDE to proving the uniqueness of the invariant law of another SDE. We prove the latter by a set of analytic tools, including the parabolic Hörmander condition and the Stroock-Varadhan support theorem.
Paper Structure (56 sections, 33 theorems, 210 equations, 3 figures, 1 table, 2 algorithms)

This paper contains 56 sections, 33 theorems, 210 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Motivating questions
  3. Precise arm-pull dynamics
  4. Adaptive inference
  5. Proof techniques
  6. Organization
  7. Notation
  8. Precise arm-pull dynamics
  9. Assumptions
  10. Sub-optimal arms
  11. Optimal arms
  12. Duality between arm 'stability' and noise interaction
  13. Application to adaptive inference
  14. Limit distribution of normalized empirical means
  15. Inference for the arm mean
  16. ...and 41 more sections

Key Result

Theorem 2.1

Suppose Assumptions assump:noise and assump:sampling_dist hold. Fix $\varepsilon \in (0,1/2)$. Then there exists some $c_0=c_0(K,\Delta,\varepsilon,\sigma,\mathscr{L}(\mathsf{Z}))>1$ such that Here recall $\bar{\Phi}^-$ is the generalized inverse of $\bar{\Phi}$.

Figures (3)

  • Figure 1: Distributional comparison between $n_{a;T}/T$ and its limit distribution $\mathbb{L}_{\lvert\mathcal{A}_0\rvert;a}^{[r],\ast}$ for an optimal arm with $K=\lvert\mathcal{A}_0\rvert=3,4$. Pink histogram: distribution of $n_{a;T}/T$ via Gaussian Thompson sampling. Blue histogram: distribution of $\mathbb{L}_{\lvert\mathcal{A}_0\rvert;a}^{[r],\ast}$ via SDE simulation.
  • Figure 2: Distributional comparison between $\mathscr{N}_K$ and $\mathcal{N}(0,1)$ for $K=3,4$ in Gaussian Thompson sampling. Pink histogram: distribution of normalized empirical means. Blue histogram: distribution of $\mathscr{N}_K$ from SDE. Red curve: distribution of $\mathcal{N}(0,1)$.
  • Figure 3: Coverage of 95% confidence intervals. Blue: CIs using critical values from $\mathscr{N}_K$. Red: CIs using critical values from $\mathcal{N}(0,1)$. Left panel: $K=4$ with two optimal arms. Right panel$K=5$ with three optimal arms.

Theorems & Definitions (58)

  • Theorem 2.1
  • Corollary 2.2
  • Remark 1
  • Definition 2.3
  • Proposition 2.4
  • Definition 2.5
  • Proposition 2.6
  • Theorem 2.7
  • Remark 2
  • Definition 2.8
  • ...and 48 more