DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shifts

TL;DR

DiSA-IQL tackles distribution shift in offline reinforcement learning for soft robot control by introducing a robustness-enhanced extension to Implicit Q-Learning. It penalizes unreliable state–action pairs via a penalty term and leverages expectile regression for stable value estimation, enabling safer generalization to unseen targets. In simulation on a soft snake target-reaching task, DiSA-IQL outperforms Behavior Cloning, Conservative Q-Learning, and vanilla IQL in both in-distribution and out-of-distribution settings, achieving higher success rates and smoother trajectories. This work advances offline RL for deformable robots and provides open-source code to support reproducibility and further research.

Abstract

Soft snake robots offer remarkable flexibility and adaptability in complex environments, yet their control remains challenging due to highly nonlinear dynamics. Existing model-based and bio-inspired controllers rely on simplified assumptions that limit performance. Deep reinforcement learning (DRL) has recently emerged as a promising alternative, but online training is often impractical because of costly and potentially damaging real-world interactions. Offline RL provides a safer option by leveraging pre-collected datasets, but it suffers from distribution shift, which degrades generalization to unseen scenarios. To overcome this challenge, we propose DiSA-IQL (Distribution-Shift-Aware Implicit Q-Learning), an extension of IQL that incorporates robustness modulation by penalizing unreliable state-action pairs to mitigate distribution shift. We evaluate DiSA-IQL on goal-reaching tasks across two settings: in-distribution and out-of-distribution evaluation. Simulation results show that DiSA-IQL consistently outperforms baseline models, including Behavior Cloning (BC), Conservative Q-Learning (CQL), and vanilla IQL, achieving higher success rates, smoother trajectories, and improved robustness. The codes are open-sourced to support reproducibility and to facilitate further research in offline RL for soft robot control.

Figures (6)

• Figure 1: (a) Soft snake robot and its pneumatic air paths for actuation. (b) Target-reaching task of the snake robot, where the RL agent drives the robot’s center of mass (COM) from the start point into the neighborhood of the target point.
• Figure 2: Illustration of goal-reaching tasks. (a) In-distribution setting, where the agent is both trained and tested in the same region (blue). (b) OOD distribution shift setting, where the agent is trained in the left half circle (blue), and evaluated on both the training region (blue) and the unseen right half circle (green).
• Figure 3: Training rewards in the (a) in-distribution setting and (b) out-of-distribution setting.
• Figure 4: Visualization of testing performance for target-reaching tasks under the in-distribution setting: (a) Behavior Cloning (BC), (b) Conservative Q-Learning (CQL), (c) Implicit Q-Learning (IQL), and (d) DiSA-IQL. Green dots represent successful target reaches, while red crosses indicate failures.
• Figure 5: Visualization of testing performance for target-reaching tasks under the out-of-distribution setting: (a) Behavior Cloning (BC), (b) Conservative Q-Learning (CQL), (c) Implicit Q-Learning (IQL), and (d) DiSA-IQL. Green dots represent successful target reaches, while red crosses indicate failures.