Asymptotically optimal reinforcement learning in Block Markov Decision Processes
Thomas van Vuren, Fiona Sloothaak, Maarten G. Wolf, Jaron Sanders
TL;DR
This work tackles reinforcement learning in Block MDPs where observations are high-dimensional but dynamics are governed by a smaller latent state space. It introduces a two-phase algorithm: first perform random exploration to recover latent structure via clustering, then apply an optimism-guided policy that exploits the identified structure. A sharp regret lower bound of $\Omega(\sqrt{A T_K}+n)$ is established for the identified class, and the proposed two-phase method achieves a matching upper bound up to polylog factors, reducing dependence on the observation count from $n^2$ to $n$ in favorable regimes and proving asymptotic optimality. The results are supported by a numerical proof of concept showing improved performance over standard UCRLVI-type methods when latent structure is present, highlighting the practical impact for large observation spaces.
Abstract
The curse of dimensionality renders Reinforcement Learning (RL) impractical in many real-world settings with exponentially large state and action spaces. Yet, many environments exhibit exploitable structure that can accelerate learning. To formalize this idea, we study RL in Block Markov Decision Processes (BMDPs). BMDPs model problems with large observation spaces, but where transition dynamics are fully determined by latent states. Recent advances in clustering methods have enabled the efficient recovery of this latent structure. However, a regret analysis that exploits these techniques to determine their impact on learning performance remained open. We are now addressing this gap by providing a regret analysis that explicitly leverages clustering, demonstrating that accurate latent state estimation can indeed effectively speed up learning. Concretely, this paper analyzes a two-phase RL algorithm for BMDPs that first learns the latent structure through random exploration and then switches to an optimism-guided strategy adapted to the uncovered structure. This algorithm achieves a regret that is $O(\sqrt{T}+n)$ on a large class of BMDPs susceptible to clustering. Here, $T$ denotes the number of time steps, $n$ is the cardinality of the observation space, and the Landau notation $O(\cdot)$ holds up to constants and polylogarithmic factors. This improves the best prior bound, $O(\sqrt{T}+n^2)$, especially when $n$ is large. Moreover, we prove that no algorithm can achieve lower regret uniformly on this same class of BMDPs. This establishes that, on this class, the algorithm achieves asymptotic optimality.