Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning

TL;DR

The paper tackles the brittleness of imitation learning in real-world robotics by leveraging non-expert data through an offline RL lens. The proposed RISE framework uses binary rewards to differentiate expert from non-expert data and employs expectile-based Q-learning, combined with Lipschitz regularization and neighborhood data augmentation to widen the policy’s effective action distribution and enable stitching back to the expert state manifold. Across sim and real-world tabletop manipulation tasks, RISE improves robustness, generalization to unseen initial conditions, and ability to incorporate varied data sources, including play data, suboptimal demonstrations, and evaluation rollouts, without requiring explicit reward annotations. The work demonstrates substantial performance gains over standard imitation and offline RL baselines, highlighting a practical pathway to robust policy learning in robotics with cheaper, heterogeneous data sources.

Abstract

Imitation learning has proven effective for training robots to perform complex tasks from expert human demonstrations. However, it remains limited by its reliance on high-quality, task-specific data, restricting adaptability to the diverse range of real-world object configurations and scenarios. In contrast, non-expert data -- such as play data, suboptimal demonstrations, partial task completions, or rollouts from suboptimal policies -- can offer broader coverage and lower collection costs. However, conventional imitation learning approaches fail to utilize this data effectively. To address these challenges, we posit that with right design decisions, offline reinforcement learning can be used as a tool to harness non-expert data to enhance the performance of imitation learning policies. We show that while standard offline RL approaches can be ineffective at actually leveraging non-expert data under the sparse data coverage settings typically encountered in the real world, simple algorithmic modifications can allow for the utilization of this data, without significant additional assumptions. Our approach shows that broadening the support of the policy distribution can allow imitation algorithms augmented by offline RL to solve tasks robustly, showing considerably enhanced recovery and generalization behavior. In manipulation tasks, these innovations significantly increase the range of initial conditions where learned policies are successful when non-expert data is incorporated. Moreover, we show that these methods are able to leverage all collected data, including partial or suboptimal demonstrations, to bolster task-directed policy performance. This underscores the importance of algorithmic techniques for using non-expert data for robust policy learning in robotics. Website: https://uwrobotlearning.github.io/RISE-offline/

Figures (10)

• Figure 1: RISE enables non-expert data to be stitched with alongside expert data to provide robust, high-coverage behavior for robotic manipulation. This allows for all collected data to be used to allow the policy to recover from novel OOD states.
• Figure 2: Various types of data on the one-leg task is shown: expert, high-coverage, and suboptimal, which can be collected by a human or from autonomous policy rollouts (for example during evaluation). RISE is able to use combinations of different expert and non-expert datasets to improve policy robustness. By stitching trajectories from non-expert data, RISE policies can succeed from much wider distributions and are robust to disturbances.
• Figure 3: Visualization of the effect of spectral norm. (a) On a planar pushing task, using IDQL naively results in an excessively narrow marginal action distribution, leading to poor performance. (b) When a spectral norm penalty is added to the behavior policy loss (Equation \ref{['eq:spectral_norm_loss']}), the action distribution is significantly widened. (c) Marginal action distributions projected onto a 1D axis are plotted alongside the learned Q function, which we empirically find to be close to the true Q function in a neighborhood of the data. Narrow action distributions often fail to encompass the optimal action (blue distribution).
• Figure 4: Depiction of tasks in sim and the real world.
• Figure 5: Generalization and Ablation (a) All experts are demonstrated from a narrow initial distribution $p_0$. We test in a larger region $p_{test}$. Our method is able to generalize to $p_{test}$ only using cheap play data. (b) Ablation of applying spectral norm regularization and data augmentation to standard IDQL on piece-assembly with high coverage. Standard offline RL (IDQL) does not stitch well, but adding our lipschitz constraint and data augmentation greatly improves performance.