Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

TL;DR

This work addresses the challenge of learning robust policies from offline data in the presence of diverse data corruption, including states, actions, rewards, and dynamics. Building on Implicit Q-Learning, the authors analyze why weighted imitation learning is naturally more robust to corruption and show that corruption introduces both imitation and heavy-tailed targets, with a theoretical bound that scales as $\mathcal{O}(\sqrt{\zeta/N})$ under mild assumptions. To tackle remaining brittleness, they propose Robust IQL (RIQL), integrating observation normalization, a Huber loss for heavy-tailed targets, and in-dataset uncertainty through quantile Q estimators, yielding strong empirical performance under random and adversarial corruption across multiple tasks. The combination of theory and extensive experiments demonstrates RIQL’s practical robustness and provides a principled path toward reliable offline RL in realistic, imperfect data settings.

Abstract

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.

Figures (18)

• Figure 1: Performance of offline RL algorithms under random attacks on the Hopper task. Many offline RL algorithms are susceptible to different types of data corruption. Notably, IQL demonstrates superior resilience to 3 out of 4 types of data corruption.
• Figure 2: Average performance of IQL w/ and w/o normalization over four datasets.
• Figure 3: (a) Heavy tailedness of Q target distributions on the Hopper task. The target values are normalized by subtracting the mean. (b) Kurtosis values of Q target distribution on Hopper and Walker tasks. The gray dashed line represents the kurtosis value of a Gaussian distribution. (c) Average final returns of IQL with different datasets. (d) Comparison of IQL w/ and w/o Huber loss.
• Figure 4: Comparison of IQL and IQL with single Q function in the (a) Hopper and (b) Walker tasks. (c) Average returns and (d) average Q values in the Hopper task under dynamics attack. IQL can be unstable due to the minimum operator.
• Figure 5: (a) Quantiles of a normal distribution. In-dataset penalty for attacked data and clean data across (b) different number of $K$ and (c) different quantile value $\alpha$.

Theorems & Definitions (8)

• Theorem 3: Robustness of IQL
• Lemma 5
• Lemma 6: Performance Difference Lemma kakade2002approximately
• proof
• proof : Proof of Theorem \ref{['thm:iql']}
• Remark 7
• proof : Proof of Lemma \ref{['lemma:huber']}
• Lemma 8: Theorem 1 of sun2020adaptive