Logarithmic Regret of Exploration in Average Reward Markov Decision Processes

TL;DR

<3-5 sentence high-level summary>Addresses regret minimization in average-reward MDPs using optimistic, model-based episodic algorithms. Proposes a Vanishing Multiplicative (VM) episode rule that improves exploration behavior while leaving Extended Value Iteration (EVI) unchanged. Establishes a coherence-based analysis linking visit rates, confidence-region dynamics, and the shrinking-shaking dichotomy to obtain logarithmic exploration regret for ergodic and communicating MDPs with prior structure, alongside minimax guarantees. Demonstrates that VM can substantially improve practical regret trajectories without sacrificing theoretical performance, and extends the framework to non-ergodic settings with priors on transition support.

Abstract

In average reward Markov decision processes, state-of-the-art algorithms for regret minimization follow a well-established framework: They are model-based, optimistic and episodic. First, they maintain a confidence region from which optimistic policies are computed using a well-known subroutine called Extended Value Iteration (EVI). Second, these policies are used over time windows called episodes, each ended by the Doubling Trick (DT) rule or a variant thereof. In this work, without modifying EVI, we show that there is a significant advantage in replacing (DT) by another simple rule, that we call the Vanishing Multiplicative (VM) rule. When managing episodes with (VM), the algorithm's regret is, both in theory and in practice, as good if not better than with (DT), while the one-shot behavior is greatly improved. More specifically, the management of bad episodes (when sub-optimal policies are being used) is much better under (VM) than (DT) by making the regret of exploration logarithmic rather than linear. These results are made possible by a new in-depth understanding of the contrasting behaviors of confidence regions during good and bad episodes.

Figures (14)

• Figure 1: The left plot displays the regret of KLUCRLfilippi_optimism_2010 over a single run with highlighted periods of sub-optimal play that are increasing in duration. In comparison, the right plot displays the regret of our proposed algorithm, where periods of sub-optimal play are much shorter resulting in a smoother regret curve.
• Figure 2: Examples of non-degeneracy (\ref{['definition_non_degeneracy']}). A degenerate Markov decision process (to the left) and a non-degenerate Markov decision process (to the right). Both models have deterministic transitions represented with arrows. Labels are reward means.
• Figure 3: An artist view of the shrinking/shaking behavior of the $\mathcal{Q}_{s,a}(t)$ as the number of new samples $N_{s,a}(t') - N_{s,a}(t) \ll N_{s,a}(t)$ increases (from dashed to solid line).
• Figure 4: Bayesian regret of UCRL2, UCRL2B and KLUCRL. Each algorithm is run 100 times on an ergodic environment with $5$ states and $2$ actions, picked at random and renewed for every run.
• Figure 5: Violin plots of the regret of KLUCRL with episodes managed by \ref{['equation_doubling_trick']} (in black with dashed lines) and by \ref{['equation_vanishing_multiplicative']} (in orange with solid lines) on a small ergodic environment. By changing \ref{['equation_doubling_trick']} to \ref{['equation_vanishing_multiplicative']}, we observe a slight improvement of the expected regret with an overall shift of its distribution to smaller values. These observations are uniform over the time horizon.

Theorems & Definitions (93)

• definition 1
• definition 2: Exploration
• definition 3: Regret of exploration
• definition 4: Non-degeneracy
• definition 5: Explorative models
• theorem 1
• theorem 2: Main result
• lemma 1: Almost-sure asymptotic regime
• definition 6: Coherence
• definition 7: Weakly regenerative episodes