Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression

Yixiu Mao, Qi Wang, Chen Chen, Yun Qu, Xiangyang Ji

TL;DR

The proposed SCAS is a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL, capable of correcting the agent from OOD states to high-value in-distribution states and suppressing OOD actions.

Abstract

In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression

TL;DR

The proposed SCAS is a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL, capable of correcting the agent from OOD states to high-value in-distribution states and suppressing OOD actions.

Abstract

In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

Paper Structure

This paper contains 49 sections, 4 theorems, 55 equations, 12 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Offline RL.
  4. OOD action issue.
  5. OOD State Correction
  6. OOD State Issue in Offline RL
  7. Origins and consequence of OOD states.
  8. OOD state correction.
  9. Limitations.
  10. Value-aware OOD State Correction
  11. Value-aware state transition.
  12. OOD state correction.
  13. Tractable optimization.
  14. Analysis of OOD Action Suppression
  15. Implementation Details
  16. ...and 34 more sections

Key Result

Proposition 1

Suppose that the environment dynamics is deterministic, then both $\bar{R}(\pi)$ and $\bar{R}_1(\pi)$ achieve their global maximum at the policy $\pi^*$, whereHere for clarity, we use the notation $M$ with slightly different meanings in different cases: in the stochastic setting, $M : \mathcal{S} \ The support of $\pi^*$ is within that of the behavior policy $\beta$: and $\pi^*$ makes the follow

Figures (12)

  • Figure 1: The resulting state distributions of offline RL algorithms and optimal values of states. (a,b,c) The state distributions generated by the learned policies of various algorithms compared with that of the offline dataset on halfcheetah-medium-expert. (d) The corresponding optimal value of each state, which is obtained by running TD3 online to convergence. SCAS-induced state distribution is almost entirely within the support of the offline distribution and avoids the low-value areas, while CQL and TD3BC tend to produce OOD states with extremely low values.
  • Figure 2: Oracle Q-values of SCAS (estimated by MC return) and learned Q-values of SCAS and other algorithms across optimization steps. Only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation (divergence).
  • Figure 3: Comparisons in the perturbed environments with varying perturbation levels. The perturbation steps are the steps of Gaussian noise added to the conducted actions in an episode. SCAS exhibits better robustness against environmental perturbations during the test phase.
  • Figure 4: Parameter study on the inverse temperature $\alpha$ and the balance coefficient $\lambda$. (a) An appropriately large $\alpha$ is crucial for achieving good performance. (b) The proposed SCAS regularization is essential and demonstrates robustness to changes in $\lambda$.
  • Figure 5: Oracle Q-values of SCAS (estimated by MC return) and learned Q-values of SCAS and other algorithms across optimization steps. Here Off-policy RL is SCAS with weight $\lambda = 0$ in Eq. \ref{['eq:PI']}. Only SCAS's OOD state correction term can achieve OOD action suppression and prevent value over-estimation (divergence).
  • ...and 7 more figures

Theorems & Definitions (7)

  • Definition 1: OOD state issue
  • Proposition 1
  • Proposition 2
  • Proposition 3: \ref{['thm:optimal deter dynamics']} in the main paper
  • proof
  • Proposition 4: \ref{['thm:optimal stoc dynamics']} in the main paper
  • proof