Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

Linjie Xu, Zhengyao Jiang, Jinyu Wang, Lei Song, Jiang Bian

TL;DR

A MCEP is proposed for test time inference with a more constrained target policy for value estimation and the empirical results show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods.

Abstract

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a \textit{Mildly Constrained Evaluation Policy (MCEP)} for test time inference with a more constrained \textit{target policy} for value estimation. Since the \textit{target policy} has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at \url{https://github.com/egg-west/MCEP.git}.

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

TL;DR

A MCEP is proposed for test time inference with a more constrained target policy for value estimation and the empirical results show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods.

Abstract

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a \textit{Mildly Constrained Evaluation Policy (MCEP)} for test time inference with a more constrained \textit{target policy} for value estimation. Since the \textit{target policy} has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at \url{https://github.com/egg-west/MCEP.git}.
Paper Structure (22 sections, 9 equations, 15 figures, 7 tables, 1 algorithm)

This paper contains 22 sections, 9 equations, 15 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Method
  5. Offline RL with mildly constrained evaluation policy
  6. Three Examples: TD3BC-MCEP, AWAC-MCEP and DQL-MCEP
  7. Experiments
  8. An illustrative example
  9. Milder constraints potentially improve performance but cause unstable learning
  10. The Evaluation policy allows milder constraints under a stable learning
  11. Performance evaluation on MuJoCo locomotion and Robotic Manipulation tasks
  12. A comparison to Inference-time action selection methods
  13. Ablation Study
  14. Conclusion
  15. Appendix
  16. ...and 7 more sections

Figures (15)

  • Figure 1: Left: diagram depicts policy trajectories for target policy $\tilde{\pi}$ and MCEP $\pi^e$. Right: policy evaluation steps to update $Q_{\tilde{\pi}}$ and policy improvement steps to update $\tilde{\pi}$ and $\pi^e$.
  • Figure 2: Evaluation of policy constraint method on a toy maze MDP \ref{['fig:toy_maze_mdp']}. In other figures, the color of a grid represents the state value and arrows indicate the actions from the corresponding policy. \ref{['fig:optimal_value']} shows the optimal value function and one optimal policy. \ref{['fig:pi_tilde']} shows a constrained policy trained from the above-mentioned offline data, with its value function calculated by $V_{\pi}=\mathbb{E}_a Q(s, \pi(a|s))$. The policy does not perform well in the low state-value area but its value function is close to the optimal value function. \ref{['fig:pi_q']} indicates that an optimal policy is recovered by deriving the greedy policy from the off-policy Q estimate (the critic).
  • Figure 3: The training process (with standard errors) of TD3BC and AWAC. Left: TD3BC on hopper-medium-v2. Middle: TD3BC on walker2d-medium-replay-v2. Right: AWAC on hopper-medium-replay-v2.
  • Figure 4: $\alpha$ values in TD3BC for value estimate and test time inference in MuJoCo locomotion tasks.
  • Figure 5: The returns with standard errors during the training on 3 humanoid tasks.
  • ...and 10 more figures