Information-directed sampling for bandits: a primer

TL;DR

This work investigates Information Directed Sampling (IDS) policies in the discounted infinite-horizon two-state Bernoulli bandit, using it as a tractable arena to contrast heuristic information-driven decisions with the optimal policy. By formulating IDS with a modified information measure and a tuning parameter α, the authors derive regret bounds and analyze IDS(α) across symmetric, one-fair-coin, and asymmetric cases. They show that in the symmetric two-state setting the optimal policy is greedy and IDS matches it, while in the one-fair-coin scenario the regret grows logarithmically with horizon, aligning with classical lower bounds. The study also explores how IDS performance depends on α and γ in the asymmetric case, identifying regimes where IDS(0) or IDS(1/2) perform better and highlighting the role of information gain in guiding exploration.

Abstract

The Multi-Armed Bandit problem provides a fundamental framework for analyzing the tension between exploration and exploitation in sequential learning. This paper explores Information Directed Sampling (IDS) policies, a class of heuristics that balance immediate regret against information gain. We focus on the tractable environment of two-state Bernoulli bandits as a minimal model to rigorously compare heuristic strategies against the optimal policy. We extend the IDS framework to the discounted infinite-horizon setting by introducing a modified information measure and a tuning parameter to modulate the decision-making behavior. We examine two specific problem classes: symmetric bandits and the scenario involving one fair coin. In the symmetric case we show that IDS achieves bounded cumulative regret, whereas in the one-fair-coin scenario the IDS policy yields a regret that scales logarithmically with the horizon, in agreement with classical asymptotic lower bounds. This work serves as a pedagogical synthesis, aiming to bridge concepts from reinforcement learning and information theory for an audience of statistical physicists.

Figures (6)

• Figure 1: (a-b) Classes of two-armed Bernoulli bandit problems. The outer square contains all two-armed bandit problems where the reward distributions are Bernoulli. The outer circle contains all the two-state problems, which in general are described by four different parameters. The inner circle contains the problems parametrized by $\theta_+,\theta_-$ -- table 1 of panel (b). The special symmetric and one fair coin cases are shown in the $(\theta_+,\theta_-)-$space and in tables 2 and 3 of panel (b), respectively. (c) Schematic sketch of a belief update for our two-state problem, where the belief is defined on a one-dimensional space, and of the evolution of a POMDP in time, showing the belief $b$, state $s$, observations $y$, and actions $a$ for two steps of the process.
• Figure 2: Examples of value functions and corresponding regrets. (a) Optimal value functions at different $\gamma$ for $\theta_+ = 0.7$ and $\theta_- = 0.55$. The dotted line corresponds to the value of the underlying Markov decision process (MDP). (b) Regrets for the values in panel (a). The dotted line corresponds to the decision boundary, where the optimal policy switches from one action to the other.
• Figure 3: Results for the symmetric bandits case. (a) Optimal regret for two probabilities of winning $\theta=0.55$ and $\theta=0.7$, and the corresponding IDS regrets. IDS achieves optimal performance in this problem. Here, $\gamma=0.99$. (b) Maximum over $\beta$ of the optimal regret as a function of $\theta$ for different values of $\gamma$. The dotted curve corresponds to the asymptotic expansion of the regret (Eq. \ref{['eq: asymptotic_symmetric_regret']} for $\gamma\to1$. (c) Maximum of the optimal regret as $\gamma\to1$, which approaches a constant value. Here, $\theta=0.55$. The dotted line is the corresponding asymptotic expansion. Here, $\theta=0.55$.
• Figure 4: Results for the one fair coin case. (a) Regret and policy for the optimal agent and IDS for two probabilities of winning of the unfair coin, $\theta_+=0.55$ and $\theta_+=0.7$. IDS($0$) performs better than IDS$(1/2$), since its decision boundary is closer to the optimal one. Here, $\gamma=0.99$.(b) Optimal regret at $\beta = 0$ as a function of $\theta_+$ for different values of $\gamma$. (c) Regret at $\beta = 0$ as $\gamma \to 1$, which diverges logarithmically. IDS($0$) behaves similarly to the optimal policy. The dotted line is the corresponding asymptotic expansion. Here, $\theta_+=0.55$.
• Figure 5: Performance of IDS($0$) and IDS($1/2$) in the asymmetric bandit problem. (a-b) Maximum relative distance to the optimal regret ($\Delta \mathcal{R}(\alpha)$) for IDS with $\alpha = 1/2$ (a) and $\alpha = 0$ (b), with $\gamma = 0.99$. The latter case performs better overall, although there are regions where the IDS($0$) policy is either too greedy or too explorative compared to the optimal one. The dashed lines correspond to the curves where $\Delta \mathcal{R}(\alpha)$ crosses a threshold of $10^{-2}$. Note that the range of $\Delta \mathcal{R}(\alpha)$ is smaller for IDS($0$). (c-d) Same, but for $\gamma = 0.999$. With respect to the previous case, the problem becomes harder, and the region where IDS($0$) is near-optimal shrinks and moves towards the axes. (e-f) For a specific choice of parameters (here $\theta_- = 0.55$), there exists an optimal value of $\alpha$ that minimizes the maximum relative regret.

Theorems & Definitions (2)

• Lemma B.1
• proof