Policy Constraint by Only Support Constraint for Offline Reinforcement Learning

TL;DR

This paper tackles offline reinforcement learning under distribution shift by addressing excessive conservatism in policy constraint methods. It proposes Only Support Constraint (OSC), a regularization that confines learning to the support of the behavior policy without penalizing actions inside the support, and uses a diffusion model to accurately estimate that support. Empirical results on D4RL benchmarks show OSC achieving state-of-the-art performance in many MuJoCo and AntMaze tasks, with ablations confirming the importance of diffusion-based support estimation and the OSC loss design. The approach reduces extrapolation errors while preserving the ability to improve beyond the behavior policy within the supported action set, offering practical improvements for offline RL deployment.

Abstract

Offline reinforcement learning (RL) aims to optimize a policy by using pre-collected datasets, to maximize cumulative rewards. However, offline reinforcement learning suffers challenges due to the distributional shift between the learned and behavior policies, leading to errors when computing Q-values for out-of-distribution (OOD) actions. To mitigate this issue, policy constraint methods aim to constrain the learned policy's distribution with the distribution of the behavior policy or confine action selection within the support of the behavior policy. However, current policy constraint methods tend to exhibit excessive conservatism, hindering the policy from further surpassing the behavior policy's performance. In this work, we present Only Support Constraint (OSC) which is derived from maximizing the total probability of learned policy in the support of behavior policy, to address the conservatism of policy constraint. OSC presents a regularization term that only restricts policies to the support without imposing extra constraints on actions within the support. Additionally, to fully harness the performance of the new policy constraints, OSC utilizes a diffusion model to effectively characterize the support of behavior policies. Experimental evaluations across a variety of offline RL benchmarks demonstrate that OSC significantly enhances performance, alleviating the challenges associated with distributional shifts and mitigating conservatism of policy constraints. Code is available at https://github.com/MoreanP/OSC.

Figures (5)

• Figure 1: Visualizing the impact of excessive conservatism. The environment is a grid task, "S" is the start location, and "G" is the goal location of the agent.
• Figure 2: We visualize the curve graph, which shows the variation of SPOT spot and OSC(ours) constraint terms with $\pi_{\beta}(a|s)$. The blue area on the coordinate axis indicates the defined support of behavior policy, $\epsilon$ is the lower bound of the support, and $\lambda$ is the constraint strength. The optimization objective of SPOT and OSC is to maximize the constraint item, that is, the constraint is in the support.
• Figure 3: Snapshots of tasks.
• Figure 4: Analyze the impact of hyperparameters on the performance of OSC on the walker2d-medium datasets. Left: With varying values of hyperparameter $\lambda$, OSC applies support constraint with different strengths. Middle: As $\alpha$ changes, the degree to which the $\sigma(x)$ function is close to the indicator function $\mathbb{I}(x)$. Right: Different $\breve{\epsilon}$ represents different defined support bounds $\{a\in A|-\log\mu(a|s)<\breve{\epsilon}\}$
• Figure 5: Degradation in normalized score of ablation methods, compared with the OSC. OSC was compared with the following ablation methods: ablation of our proposed loss function, using SPOT loss function and diffusion density estimator; ablation of diffusion density estimator, using OSC loss and CVAE estimator; simultaneous ablation of OSC loss and diffusion estimator which is using SPOT loss and CVAE. hc=HalfCheetah, ho=Hopper, w=Walker2d, me=medium-expert, m=medium and mr=medium-replay.