MoMA: Model-based Mirror Ascent for Offline Reinforcement Learning
Mao Hong, Zhiyue Zhang, Yue Wu, Yanxun Xu
TL;DR
MoMA tackles offline reinforcement learning by combining model-based pessimism with mirror ascent, enabling unrestricted policy spaces within partial data coverage. It decouples conservative policy evaluation from policy improvement, using a confidence-set of transition models to obtain pessimistic Q-values and a MA-based update to optimize policies with general function approximations. Theoretical guarantees bound suboptimality by separating model error, optimization error, and function-approximation error while avoiding policy-class size dependence. A practical algorithm via primal-dual optimization and function approximation is proposed, together with continuous-action extensions and empirical validation on synthetic data and D4RL benchmarks. Overall, MoMA delivers a principled, efficient, and flexible framework for offline RL with strong theoretical and empirical performance advantages.
Abstract
Model-based offline reinforcement learning methods (RL) have achieved state-of-the-art performance in many decision-making problems thanks to their sample efficiency and generalizability. Despite these advancements, existing model-based offline RL approaches either focus on theoretical studies without developing practical algorithms or rely on a restricted parametric policy space, thus not fully leveraging the advantages of an unrestricted policy space inherent to model-based methods. To address this limitation, we develop MoMA, a model-based mirror ascent algorithm with general function approximations under partial coverage of offline data. MoMA distinguishes itself from existing literature by employing an unrestricted policy class. In each iteration, MoMA conservatively estimates the value function by a minimization procedure within a confidence set of transition models in the policy evaluation step, then updates the policy with general function approximations instead of commonly-used parametric policy classes in the policy improvement step. Under some mild assumptions, we establish theoretical guarantees of MoMA by proving an upper bound on the suboptimality of the returned policy. We also provide a practically implementable, approximate version of the algorithm. The effectiveness of MoMA is demonstrated via numerical studies.