Non-Linear Reinforcement Learning in Large Action Spaces: Structural Conditions and Sample-efficiency of Posterior Sampling
Alekh Agarwal, Tong Zhang
TL;DR
This work addresses sample-efficient reinforcement learning in environments with large or continuous action spaces using non-linear function approximation. It introduces TS^3, a two-time-scale posterior-sampling algorithm that uses a generalized Bellman-rank framework and a linear embedding of Bellman errors to enable efficient exploration and learning across infinite actions. The authors prove regret bounds that scale with the generalized Bellman rank br(ε), the embedding dimension dc(ε), horizon H, and function-class complexity, unifying linear MDPs and finite-action problems under a common framework. They also propose TS^2-D for known linear embeddings, achieving faster 1/ε^2 rates via experimental design, and discuss implications for combinatorial action spaces and mixtures of MDPs. Overall, the paper advances representation learning in RL with continuous actions by marrying posterior sampling with structured decompositions of Bellman errors, while outlining avenues to improve rates and broaden applicability.
Abstract
Provably sample-efficient Reinforcement Learning (RL) with rich observations and function approximation has witnessed tremendous recent progress, particularly when the underlying function approximators are linear. In this linear regime, computationally and statistically efficient methods exist where the potentially infinite state and action spaces can be captured through a known feature embedding, with the sample complexity scaling with the (intrinsic) dimension of these features. When the action space is finite, significantly more sophisticated results allow non-linear function approximation under appropriate structural constraints on the underlying RL problem, permitting for instance, the learning of good features instead of assuming access to them. In this work, we present the first result for non-linear function approximation which holds for general action spaces under a linear embeddability condition, which generalizes all linear and finite action settings. We design a novel optimistic posterior sampling strategy, TS^3 for such problems, and show worst case sample complexity guarantees that scale with a rank parameter of the RL problem, the linear embedding dimension introduced in this work and standard measures of the function class complexity.