Dynamic Decision-Making under Model Misspecification: A Stochastic Stability Approach

TL;DR

This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified, and offers the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics.

Abstract

Dynamic decision-making under model uncertainty is central to many economic environments, yet existing bandit and reinforcement learning algorithms rely on the assumption of correct model specification. This paper studies the behavior and performance of one of the most commonly used Bayesian reinforcement learning algorithms, Thompson Sampling (TS), when the model class is misspecified. We first provide a complete dynamic classification of posterior evolution in a misspecified two-armed Gaussian bandit, identifying distinct regimes: correct model concentration, incorrect model concentration, and persistent belief mixing, characterized by the direction of statistical evidence and the model-action mapping. These regimes yield sharp predictions for limiting beliefs, action frequencies, and asymptotic regret. We then extend the analysis to a general finite model class and develop a unified stochastic stability framework that represents posterior evolution as a Markov process on the belief simplex. This approach characterizes two sufficient conditions to classify the ergodic and transient behaviors and provides inductive dimensional reductions of the posterior dynamics. Our results offer the first qualitative and geometric classification of TS under misspecification, bridging Bayesian learning with evolutionary dynamics, and also build the foundations of robust decision-making in structured bandits.

Figures (8)

• Figure 1: Crossing configuration showing misspecified models.
• Figure 2: Comparison of Case 1 and Case 2 scenarios. In Case 1, both models agree arm $A_1$ is optimal. In Case 2, models disagree: $\nu$ prefers $A_1$ while $\gamma$ prefers $A_2$. The three regions (I, II, III) for the true expected rewards are indicated on each action line. The region
• Figure 3: Vector field visualizations. Left two columns show S-space; right two columns show simplex space. (a,b) Self-confirming equilibrium; (c,d) Self-defeating equilibrium. (e,f) Drift towards face interior; (g,h) Drift towards vertex.
• Figure 4: Different equilibrium types in multi-model Thompson Sampling. In each panel, the arrows $d(a_1), d(a_2), d(a_3)$ represent the drift vectors associated with each model, which are the expected directions in which log-odds would move if that model were played with probability one. The shaded region is the convex hull $\mathrm{conv}\{d(a_1), d(a_2), d(a_3)\}$, representing all possible mean drift directions achievable by mixing over models. We list out 4 representative cases: (a) Interior equilibrium with self-confirming models. (b) Interior equilibrium with self-defeating models. (c) No equilibria: drift towards face interior. (d) No equilibria: drift towards vertex.
• Figure 5: Agreement case simulation results.

Theorems & Definitions (46)

• Definition 1: Model Misspecification
• Theorem 1: Thompson Sampling: Agreement Case
• Theorem 2: Thompson Sampling: Self-Confirming Case
• Remark 1: Attractors and path dependence
• Corollary 1: Self-confirming limits are strict Berk--Nash equilibria
• Theorem 3: Thompson Sampling: Uniform Dominance Case
• Theorem 4: Thompson Sampling: Self-Defeating Case
• Corollary 2: Non-concentration and persistent experimentation
• Lemma 1: Irreducibility and Aperiodicity of the Log-Odds Process
• Lemma 2: Dichotomy of the Log-Odds Process