The regret lower bound for communicating Markov Decision Processes

TL;DR

The paper tackles the problem of instance-dependent regret lower bounds for communicating Markov decision processes under average reward. It introduces a three-fold mechanism—mandatory exploration, mandatory co-exploration, and navigation constraints—captured via an optimization over invariant measures to define the bound $K_\star(\mathbf{M})$. The lower bound is shown to be computationally hard (Σ$_2^P$-hard; testing feasibility is coNP-hard), and the authors provide a constructive method to approximate it through local environment modifications, with several classical MDP settings recovering known results. The work also formalizes the role of minors (contracted invariant measures) in understanding second-order navigation effects and proves that navigation constraints are essential for tight bounds. Overall, the paper lays foundational theory for model-dependent regret in general MDPs, linking information-theoretic constraints with environmental structure and opening avenues for approximation and extension to broader settings.

Abstract

This paper is devoted to the extension of the regret lower bound beyond ergodic Markov decision processes (MDPs) in the problem dependent setting. While the regret lower bound for ergodic MDPs is well-known and reached by tractable algorithms, we prove that the regret lower bound becomes significatively more complex in communicating MDPs. Our lower bound revisits the necessary explorative behavior of consistent learning agents and further explains that all optimal regions of the environment must be overvisited compared to sub-optimal ones, a phenomenon that we refer to as co-exploration. In tandem, we show that these two explorative and co-explorative behaviors are intertwined with navigation constraints obtained by scrutinizing the navigation structure at logarithmic scale. The resulting lower bound is expressed as the solution of an optimization problem that, in many standard classes of MDPs, can be specialized to recover existing results. From a computational perspective, it is provably $Σ_2^\textrm{P}$-hard in general and as a matter of fact, even testing the membership to the feasible region is coNP-hard. We further provide an algorithm to approximate the lower bound in a constructive way.

Figures (7)

• Figure 1: Two 10 state Markov decision processes $\mathbf{M}$ and $\mathbf{M}^\dagger$ with deterministic transitions. Every arrow is a choice of action that leads to the same pointed state with probability one and with reward $\mathrm{N}(x, 1)$ where $x \in \{*\}{0, 1, 2}$ is the label of the arrow.
• Figure 2: The contraction of $\mathbf{M}$ by $\mathcal{X}_{\star}(\mathbf{M})$. The states spawning $\mathcal{X}_{\star}(\mathbf{M})$ are merged together into a single state, forming a new Markov decision process $\mathbf{M}/\mathcal{X}_{\star}(\mathbf{M})$, where the navigational behavior of a consistent learner is better understood.
• Figure 3: An example of a two-state deterministic Markov decision process with Bernoulli rewards where the navigation constraints are necessary to obtain the right regret lower bound.
• Figure 4: The ( Choose $k$) widget, where $\sigma_k^2 := \frac{\sigma^2}{w_k}$. The labels $\mathrm{N}(x, y)\$$ are the (Gaussian) reward distributions.
• Figure 5: Embedding a knapsack problem in a Markov decision process.

Theorems & Definitions (69)

• Definition 1: Regret, auer_near_optimal_2009
• Proposition 1
• proof
• Definition 2: Consistency
• Definition 3: Classification of pairs
• Lemma 2
• proof
• Definition 4: Confusing models
• Definition 5
• Theorem 3: Regret lower bound