Unified Algorithms for RL with Decision-Estimation Coefficients: PAC, Reward-Free, Preference-Based Learning, and Beyond
Fan Chen, Song Mei, Yu Bai
TL;DR
This work proposes a unified DEC-based framework for reinforcement learning that extends beyond traditional no-regret objectives to encompass PAC learning, reward-free exploration, all-policy model estimation, and preference-based RL. Central to the framework is the generalized Decision-Estimation Coefficient ($ abla$G-DEC) and a corresponding meta-algorithm ($ abla$G-E2D), which instantiate goal-specific bounds by tuning a divergence-based trade-off between information gain and suboptimality. The paper provides PAC and regret guarantees, lower bounds, and concrete instantiations (RFDEC, AMDEC, PBDEC), all under the umbrella of decouplable representations that unify several existing structural conditions. It also connects these DEC-based guarantees to optimistic model-based methods such as MOPS and OMLE, showing that they admit similar sample complexity bounds under shared structural assumptions. The results recover prior findings and yield new guarantees across a wide range of model classes, including reward-free exploration, partially observable RL, and multi-agent settings, while highlighting directions for computational efficiency and potential gaps between bounds. Overall, the work advances a cohesive, theory-driven path toward understanding and achieving sample-efficient RL across diverse learning goals with a single unified framework.
Abstract
Modern Reinforcement Learning (RL) is more than just learning the optimal policy; Alternative learning goals such as exploring the environment, estimating the underlying model, and learning from preference feedback are all of practical importance. While provably sample-efficient algorithms for each specific goal have been proposed, these algorithms often depend strongly on the particular learning goal and thus admit different structures correspondingly. It is an urging open question whether these learning goals can rather be tackled by a single unified algorithm. We make progress on this question by developing a unified algorithm framework for a large class of learning goals, building on the Decision-Estimation Coefficient (DEC) framework. Our framework handles many learning goals such as no-regret RL, PAC RL, reward-free learning, model estimation, and preference-based learning, all by simply instantiating the same generic complexity measure called "Generalized DEC", and a corresponding generic algorithm. The generalized DEC also yields a sample complexity lower bound for each specific learning goal. As applications, we propose "decouplable representation" as a natural sufficient condition for bounding generalized DECs, and use it to obtain many new sample-efficient results (and recover existing results) for a wide range of learning goals and problem classes as direct corollaries. Finally, as a connection, we re-analyze two existing optimistic model-based algorithms based on Posterior Sampling and Maximum Likelihood Estimation, showing that they enjoy sample complexity bounds under similar structural conditions as the DEC.