Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient and Interpretable Bandit Algorithms

Subhojyoti Mukherjee, Ruihao Zhu, Branislav Kveton

TL;DR

This work addresses the need for interpretable exploration in bandits by introducing CODE, a Constrained Optimal Design–based algorithm that explores among plausible actions to maximally reduce uncertainty about the unknown parameter. CODE replaces phased exploration with a per-round constraint and a D-optimal design objective, yielding near-optimal regret in both $K$-armed and linear bandits while enabling changing action sets; it also introduces a novel interpretability metric $Q_n$ that measures cumulative model uncertainty. Theoretical results show $R_n = O\left(\frac{K \log n}{\Delta_{\min}}\right)$ in the $K$-armed case and $R_n = \tilde{O}(d \sqrt{n})$ in linear bandits, along with a bound $Q_n = O(K \log(1/\delta) \log n)$ and $Q_n = O(K \log(1/\delta) \log n)$ in the respective settings. Empirical results on synthetic data and real datasets demonstrate CODE's ability to outperform other interpretable designs while matching the performance of popular uninterpretable methods, indicating meaningful practical impact for explainable sequential decision-making.

Abstract

Motivated by the importance of explainability in modern machine learning, we design bandit algorithms that are efficient and interpretable. A bandit algorithm is interpretable if it explores with the objective of reducing uncertainty in the unknown model parameter. To quantify the interpretability, we introduce a novel metric of model error, which compares the rate reduction of the mean reward estimates to their actual means among all the plausible actions. We propose CODE, a bandit algorithm based on a Constrained Optimal DEsign, that is interpretable and maximally reduces the uncertainty. The key idea in CODE is to explore among all plausible actions, determined by a statistical constraint, to achieve interpretability. We implement CODE efficiently in both multi-armed and linear bandits and derive near-optimal regret bounds by leveraging the optimality criteria of the approximate optimal design. CODE can be also viewed as removing phases in conventional phased elimination, which makes it more practical and general. We demonstrate the advantage of CODE by numerical experiments on both synthetic and real-world problems. CODE outperforms other state-of-the-art interpretable designs while matching the performance of popular but uninterpretable designs, such as upper confidence bound algorithms.

Efficient and Interpretable Bandit Algorithms

TL;DR

This work addresses the need for interpretable exploration in bandits by introducing CODE, a Constrained Optimal Design–based algorithm that explores among plausible actions to maximally reduce uncertainty about the unknown parameter. CODE replaces phased exploration with a per-round constraint and a D-optimal design objective, yielding near-optimal regret in both -armed and linear bandits while enabling changing action sets; it also introduces a novel interpretability metric that measures cumulative model uncertainty. Theoretical results show in the -armed case and in linear bandits, along with a bound and in the respective settings. Empirical results on synthetic data and real datasets demonstrate CODE's ability to outperform other interpretable designs while matching the performance of popular uninterpretable methods, indicating meaningful practical impact for explainable sequential decision-making.

Abstract

Motivated by the importance of explainability in modern machine learning, we design bandit algorithms that are efficient and interpretable. A bandit algorithm is interpretable if it explores with the objective of reducing uncertainty in the unknown model parameter. To quantify the interpretability, we introduce a novel metric of model error, which compares the rate reduction of the mean reward estimates to their actual means among all the plausible actions. We propose CODE, a bandit algorithm based on a Constrained Optimal DEsign, that is interpretable and maximally reduces the uncertainty. The key idea in CODE is to explore among all plausible actions, determined by a statistical constraint, to achieve interpretability. We implement CODE efficiently in both multi-armed and linear bandits and derive near-optimal regret bounds by leveraging the optimality criteria of the approximate optimal design. CODE can be also viewed as removing phases in conventional phased elimination, which makes it more practical and general. We demonstrate the advantage of CODE by numerical experiments on both synthetic and real-world problems. CODE outperforms other state-of-the-art interpretable designs while matching the performance of popular but uninterpretable designs, such as upper confidence bound algorithms.
Paper Structure (22 sections, 11 theorems, 43 equations, 5 figures, 1 table, 1 algorithm)

This paper contains 22 sections, 11 theorems, 43 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Interpretable Exploration via Constrained Optimal Design
  4. Explainability Metric
  5. Related Optimal Design Works
  6. $K$-Armed Bandits: A Warm-Up
  7. Regret Analysis
  8. Explainability Metric
  9. Linear Bandits
  10. Regret Analysis
  11. Experiments
  12. Linear Bandits
  13. Linear Bandits with Changing Arms
  14. MovieLens
  15. White Wine Quality
  16. ...and 7 more sections

Key Result

Theorem 1

The regret of CODE in $K$-armed bandits is $R_n = O\left(\dfrac{K\log n}{\Delta_{\min}}\right)$, where $\Delta_{\min} \coloneqq \min_{a \in \mathcal{A}} \Delta(a)$.

Figures (5)

  • Figure 1: (a) Regret of CODE and five baselines in linear bandits, reported as a function of horizon $n$. (b) Interpretability of CODE and the baselines.
  • Figure 2: (a) Regret of CODE and five baselines in linear bandits, reported as a function of horizon $n$. (b) Interpretability of CODE and the baselines.
  • Figure 3: (a) Regret of CODE and five baselines on the MovieLens problem, reported as a function of horizon $n$. (b) Interpretability of CODE and the baselines.
  • Figure 4: (a) Regret of CODE and five baselines on the Wine Quality dataset, shown as a function of horizon $n$. (b) Interpretability of CODE and the baselines.
  • Figure 5: (a) Regret of CODE and five baselines on the Heart Failure dataset, shown as a function of horizon $n$. (b) Interpretability of CODE and the baselines.

Theorems & Definitions (17)

  • Theorem 1
  • Lemma 4.1
  • Lemma 4.2
  • proof
  • Lemma 4.3
  • proof
  • Theorem 2
  • Lemma 5.1
  • Lemma 5.2
  • Lemma 1
  • ...and 7 more