Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models

TL;DR

The paper addresses data inefficiency in reinforcement learning by exploiting prior model knowledge to bound the optimal Q-function over a model set that contains the true environment. It introduces BUMEX, a bounded-uncertainty, model-based exploration strategy that derives optimistic/pessimistic Q-function bounds using contraction operators $\\underaccent{\\bar}{\\mathcal T}$ and $\\bar{\\mathcal T}$, and augments this with a data-regularized model-set optimization to guarantee convergence. In finite state-action settings, the authors provide a practical algorithm within the BMDP framework, prove finite-time convergence under mild conditions, and show empirical gains on Frozen Lake, Cartpole, and Taxi; a toolbox is released for public use. Overall, the work offers a principled way to accelerate RL by leveraging known structure and observed data to tighten value-function bounds and steer exploration toward informative regions of the space.

Abstract

Reinforcement learning (RL) is a powerful framework for decision-making in uncertain environments, but it often requires large amounts of data to learn an optimal policy. We address this challenge by incorporating prior model knowledge to guide exploration and accelerate the learning process. Specifically, we assume access to a model set that contains the true transition kernel and reward function. We optimize over this model set to obtain upper and lower bounds on the Q-function, which are then used to guide the exploration of the agent. We provide theoretical guarantees on the convergence of the Q-function to the optimal Q-function under the proposed class of exploring policies. Furthermore, we also introduce a data-driven regularized version of the model set optimization problem that ensures the convergence of the class of exploring policies to the optimal policy. Lastly, we show that when the model set has a specific structure, namely the bounded-parameter MDP (BMDP) framework, the regularized model set optimization problem becomes convex and simple to implement. In this setting, we also prove finite-time convergence to the optimal policy under mild assumptions. We demonstrate the effectiveness of the proposed exploration strategy, which we call BUMEX (Bounded Uncertainty Model-based Exploration), in a simulation study. The results indicate that the proposed method can significantly accelerate learning in benchmark examples. A toolbox is available at https://github.com/JvHulst/BUMEX.

Figures (2)

• Figure 1: Schematic representation of Q-function bounds. From this particular example, using Proposition \ref{['prop:suboptimality_inputs']} we can deduce that the choice $u=3$ is guaranteed to be suboptimal for this state $x$, as its upper bound Q-value $\bar{Q}(x,3)$ is lower than $\underaccent{\bar{}}{V}(x)$. By \ref{['eq:exploring_policy_weights']}, the input choice $u=3$ is therefore assigned a weight of $\zeta$. In contrast, actions $u=1$ and $u=2$, for which the bounds do not decisively rule out optimality, receive weight $\beta(x,u)$. Since $\underaccent{\bar{}}{Q}(x,1)$ is below $\underaccent{\bar{}}{V}(x)$, the proposed heuristics in Section \ref{['subsec:BO_weighting']} explore the input $u=2$ more frequently.
• Figure 2: Evaluation performance of BUMEX versus standard $\epsilon$-greedy Q-learning, Q-learning with UCB exploration, UCRL2, and PSRL across three benchmark environments. Plotted are the median, and the 25th and 75th percentiles of 100 Monte Carlo runs, where the performance is evaluated every 25 training episodes (frozen lake) or 100 training episodes (cartpole, taxi) by averaging the returns obtained by applying the greedy policy for 100 episodes.

Theorems & Definitions (11)

• Lemma 2
• proof
• Proposition 3
• Theorem 4
• proof
• Proposition 5
• proof
• Theorem 6
• proof
• Lemma 8