Gap-Dependent Bounds for Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximation

TL;DR

This work provides the first gap-dependent regret bound for the nearly minimax-optimal algorithm LSVI-UCB++ and introduces a concurrent variant that enables efficient parallel exploration across multiple agents and establishes the first gap-dependent sample complexity upper bound for online multi-agent RL with linear function approximation.

Abstract

We study gap-dependent performance guarantees for nearly minimax-optimal algorithms in reinforcement learning with linear function approximation. While prior works have established gap-dependent regret bounds in this setting, existing analyses do not apply to algorithms that achieve the nearly minimax-optimal worst-case regret bound $\tilde{O}(d\sqrt{H^3K})$, where $d$ is the feature dimension, $H$ is the horizon length, and $K$ is the number of episodes. We bridge this gap by providing the first gap-dependent regret bound for the nearly minimax-optimal algorithm LSVI-UCB++ (He et al., 2023). Our analysis yields improved dependencies on both $d$ and $H$ compared to previous gap-dependent results. Moreover, leveraging the low policy-switching property of LSVI-UCB++, we introduce a concurrent variant that enables efficient parallel exploration across multiple agents and establish the first gap-dependent sample complexity upper bound for online multi-agent RL with linear function approximation, achieving linear speedup with respect to the number of agents.

Theorems & Definitions (24)

• Definition 3.1
• Definition 3.2
• Definition 3.3
• Proposition 3.4: Proposition 3.3 of he2021logarithmic
• Theorem 4.1
• Corollary 4.2
• Theorem 4.3
• Lemma 5.1
• Lemma 5.2: Informal
• Lemma 5.3