VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

TL;DR

VIPO tackles offline reinforcement learning by penalizing the inconsistency between a value function learned from the fixed dataset and one learned from a dynamics model. This self-supervised signal guides the model toward more accurate dynamics, improving data efficiency and generalization without relying on expensive architectures. The authors derive a gradient expression for the augmented loss and implement a practical algorithm that jointly learns two value estimates, trains an ensemble of models, and uses short-rollout data with a SAC-based planner. Empirically, VIPO achieves state-of-the-art performance on D4RL and NeoRL benchmarks, demonstrates more faithful uncertainty estimation, and reduces predictive error compared to prior model-based offline RL methods, highlighting its potential as a general enhancement for existing approaches.

Abstract

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the one estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy.

Figures (4)

• Figure 1: Comparison of VIPO with previous model-based approaches for learning a pessimistic dynamics model. (A) Previous model-based methods make a single use of data to learn an ensemble of models and then use its uncertainty to apply ad-hoc, pessimistic adjustments to the model predictions. (B) VIPO leverages the data in two ways: (1) it learns a value function $V_d^\mu(s)$ directly by minimizing the mean-square Bellman error (MSBE) loss; (2) it first learns the dynamics model by minimizing the negative log-likelihood (NLL) loss and then estimate the ensemble value functions $V_m^\mu(s)$. VIPO utilizes the discrepancy between these two types of value functions as an additional self-supervised loss to improve the model learning performance.
• Figure 2: Uncertainty of models trained by MOPO and VIPO, averaged over 4 random seeds.
• Figure 3: The model prediction error on four walker2d datasets of the D4RL task: (a) walker2d-random-v2; (b) walker2d-medium-v2; (c) walker2d-medium-replay-v2; (d) walker2d-medium-expert-v2.
• Figure 4: The policy training process on four walker2d datasets of the D4RLtask: (a) walker2d-random-v2; (b) walker2d-medium-v2; (c) walker2d-medium-replay-v2; (d) walker2d-medium-expert-v2.

Theorems & Definitions (6)

• Proposition 3.1
• Proposition 3.2
• Theorem 3.3: Model Gradient Theorem
• proof
• proof
• proof