Automatic Constraint Policy Optimization based on Continuous Constraint Interpolation Framework for Offline Reinforcement Learning

TL;DR

This work tackles extrapolation error in offline reinforcement learning by unifying three policy-constraint families—weighted behavior cloning, KL-based density regularization, and support constraints—under a single Continuous Constraint Interpolation framework governed by a scalar parameter $\lambda$. It introduces Automatic Constraint Policy Optimization (ACPO), a primal–dual algorithm that automatically tunes $\lambda$ during training and derives a closed-form non-parametric policy $\pi^*_{\lambda}$ whose density couples the Q-function with the behavior policy. The authors prove a maximum-entropy performance difference lemma and provide lower bounds on the performance of both the optimal and parametric policies, linking these guarantees to distributional shift via total variation. Empirically, ACPO achieves robust improvements on the D4RL and NeoRL2 benchmarks and demonstrates useful lambda-dynamics and ablations that illuminate the roles of constraint regimes and behavior-density estimation.

Abstract

Offline Reinforcement Learning (RL) relies on policy constraints to mitigate extrapolation error, where both the constraint form and constraint strength critically shape performance. However, most existing methods commit to a single constraint family: weighted behavior cloning, density regularization, or support constraints, without a unified principle that explains their connections or trade-offs. In this work, we propose Continuous Constraint Interpolation (CCI), a unified optimization framework in which these three constraint families arise as special cases along a common constraint spectrum. The CCI framework introduces a single interpolation parameter that enables smooth transitions and principled combinations across constraint types. Building on CCI, we develop Automatic Constraint Policy Optimization (ACPO), a practical primal--dual algorithm that adapts the interpolation parameter via a Lagrangian dual update. Moreover, we establish a maximum-entropy performance difference lemma and derive performance lower bounds for both the closed-form optimal policy and its parametric projection. Experiments on D4RL and NeoRL2 demonstrate robust gains across diverse domains, achieving state-of-the-art performance overall.

Figures (6)

• Figure 1: Special-case comparisons and the evolution of $\lambda$ learned by ACPO during training.
• Figure 2: Normalized score differences on Gym-MuJoCo datasets.
• Figure 3: Distributions of $\log \pi_\beta(a|s)$ under Gaussian and CVAE behavior estimators on in-dataset actions and OOD actions.
• Figure 4: Gaussian and CVAE behavior models under $\xi \in \{0.1, 0.5, 0.8\}$.
• Figure 5: D4RL Learning Curves.

Theorems & Definitions (12)

• Proposition 4.1
• Lemma 4.2: Maximum Entropy Performance Difference
• Theorem 4.3: Performance Lower Bound for the Optimal Policy
• Theorem 4.4: Performance Lower Bound for a Parametric Policy
• Proposition 2.1
• proof
• Lemma 3.1: Maximum-Entropy Performance Difference Lemma
• proof
• Theorem 3.2: Performance Lower Bound for the Optimal Policy
• proof