Adaptive Resolving Methods for Reinforcement Learning with Function Approximations
Jiashuo Jiang, Yiming Zong, Yinyu Ye
TL;DR
The paper addresses RL in settings with large or infinite state-action spaces by leveraging function approximation and an LP-based reformulation. It introduces an online resolving framework that adaptively updates a reduced LP whose basis is identified from data, achieving a problem-dependent $ ilde{O}(1/N)$ suboptimality gap rather than the standard worst-case $O(1/\\sqrt{N})$. The key contributions include reducing the LP to a basis-aligned dimension $d_2$ and solving via a two-phase resolving procedure that remains robust under degeneracy and estimation error, yielding a rigorous instance-dependent sample complexity bound. Empirically, the method demonstrates efficient performance and competitive or superior results compared to baselines on the Mountain Car task, while maintaining strong constraint satisfaction despite noisy data.
Abstract
Reinforcement learning (RL) problems are fundamental in online decision-making and have been instrumental in finding an optimal policy for Markov decision processes (MDPs). Function approximations are usually deployed to handle large or infinite state-action space. In our work, we consider the RL problems with function approximation and we develop a new algorithm to solve it efficiently. Our algorithm is based on the linear programming (LP) reformulation and it resolves the LP at each iteration improved with new data arrival. Such a resolving scheme enables our algorithm to achieve an instance-dependent sample complexity guarantee, more precisely, when we have $N$ data, the output of our algorithm enjoys an instance-dependent $\tilde{O}(1/N)$ suboptimality gap. In comparison to the $O(1/\sqrt{N})$ worst-case guarantee established in the previous literature, our instance-dependent guarantee is tighter when the underlying instance is favorable, and the numerical experiments also reveal the efficient empirical performances of our algorithms.