Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting to Stochastic and Adversarial Losses in Episodic MDPs with Aggregate Bandit Feedback

Shinji Ito, Kevin Jamieson, Haipeng Luo, Arnab Maiti, Taira Tsuchiya

TL;DR

The paper addresses online learning in finite-horizon episodic MDPs under aggregate bandit feedback, where only episode-level losses are observed. It introduces best-of-both-worlds algorithms that achieve $O(\log T)$ regret in stochastic settings and $\tilde{O}(\sqrt{T})$ regret in adversarial settings, both for known-transition and unknown-transition cases. The core methodology combines FTRL over occupancy measures with carefully designed loss estimators that work with aggregate feedback and have controlled second moments, leveraging ideas from online shortest path problems. The results include near-optimal regret bounds, gap-dependent lower bounds, robustness to corruption, and extensions to the shortest path problem, with implications for practical learning under trajectory-level feedback. The work also outlines limitations and future directions toward policy-optimization approaches and broader function-approximation settings beyond tabular MDPs.

Abstract

We study online learning in finite-horizon episodic Markov decision processes (MDPs) under the challenging aggregate bandit feedback model, where the learner observes only the cumulative loss incurred in each episode, rather than individual losses at each state-action pair. While prior work in this setting has focused exclusively on worst-case analysis, we initiate the study of best-of-both-worlds (BOBW) algorithms that achieve low regret in both stochastic and adversarial environments. We propose the first BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback. In the case of known transitions, our algorithms achieve $O(\log T)$ regret in stochastic settings and ${O}(\sqrt{T})$ regret in adversarial ones. Importantly, we also establish matching lower bounds, showing the optimality of our algorithms in this setting. We further extend our approach to unknown-transition settings by incorporating confidence-based techniques. Our results rely on a combination of FTRL over occupancy measures, self-bounding techniques, and new loss estimators inspired by recent advances in online shortest path problems. Along the way, we also provide the first individual-gap-dependent lower bounds and demonstrate near-optimal BOBW algorithms for shortest path problems with bandit feedback.

Adapting to Stochastic and Adversarial Losses in Episodic MDPs with Aggregate Bandit Feedback

TL;DR

The paper addresses online learning in finite-horizon episodic MDPs under aggregate bandit feedback, where only episode-level losses are observed. It introduces best-of-both-worlds algorithms that achieve regret in stochastic settings and regret in adversarial settings, both for known-transition and unknown-transition cases. The core methodology combines FTRL over occupancy measures with carefully designed loss estimators that work with aggregate feedback and have controlled second moments, leveraging ideas from online shortest path problems. The results include near-optimal regret bounds, gap-dependent lower bounds, robustness to corruption, and extensions to the shortest path problem, with implications for practical learning under trajectory-level feedback. The work also outlines limitations and future directions toward policy-optimization approaches and broader function-approximation settings beyond tabular MDPs.

Abstract

We study online learning in finite-horizon episodic Markov decision processes (MDPs) under the challenging aggregate bandit feedback model, where the learner observes only the cumulative loss incurred in each episode, rather than individual losses at each state-action pair. While prior work in this setting has focused exclusively on worst-case analysis, we initiate the study of best-of-both-worlds (BOBW) algorithms that achieve low regret in both stochastic and adversarial environments. We propose the first BOBW algorithms for episodic tabular MDPs with aggregate bandit feedback. In the case of known transitions, our algorithms achieve regret in stochastic settings and regret in adversarial ones. Importantly, we also establish matching lower bounds, showing the optimality of our algorithms in this setting. We further extend our approach to unknown-transition settings by incorporating confidence-based techniques. Our results rely on a combination of FTRL over occupancy measures, self-bounding techniques, and new loss estimators inspired by recent advances in online shortest path problems. Along the way, we also provide the first individual-gap-dependent lower bounds and demonstrate near-optimal BOBW algorithms for shortest path problems with bandit feedback.
Paper Structure (49 sections, 52 theorems, 220 equations, 3 tables, 1 algorithm)

This paper contains 49 sections, 52 theorems, 220 equations, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution
  3. Problem setup
  4. Episodic Markov decision process
  5. Regime of environments
  6. Online shortest path problem
  7. Core idea: construction of loss estimator with aggregate feedback
  8. Review of the approach of maiti2025efficient for the online shortest path problem
  9. Loss estimator for MDPs with known transition
  10. Loss estimator for MDPs with unknown transition
  11. Second moment of loss estimators
  12. Algorithm and regret bounds
  13. Warmup: online shortest path problem
  14. MDPs with known transition
  15. MDPs with unknown transition
  16. ...and 34 more sections

Key Result

Lemma 1

Suppose $\bar{\ell}$ is defined by $\bar{\ell}(s,a) = Q^{\pi}(s,a;\ell) - V^{\pi}(s;\ell)$ for some $\pi \in \Pi$ and for all $s \in S$ and $a \in A$. We then have for any $\pi' \in \Pi$, $s \in S$ and $a \in A$.

Theorems & Definitions (88)

  • Lemma 1
  • Definition 1: self-bounding regime for MDPs
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Theorem 1
  • Corollary 1
  • Corollary 2
  • Theorem 2
  • Theorem 3
  • ...and 78 more