DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control

TL;DR

This work addresses robust sim2real transfer for robot control under stochastic delays and disturbances by introducing the disturbance-augmented MDP ($\text{DAMDP}$) and Disturbance-Aware RL (DiAReL). It trains a disturbance observer (DOB) alongside an inverse dynamics model to augment observations with a history of estimated disturbances, enabling delay-resolved policies to withstand variations in dynamics. Across reaching and pushing tasks, disturbance-augmented policies outperform disturbance-unaware baselines in stabilization and robustness, both in simulation and on a real robot, narrowing the sim2real gap. The approach offers a principled, data-driven pathway to integrate disturbance estimation with delay-aware reinforcement learning for robust robotic manipulation.

Abstract

Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning algorithms train policies to learn optimal interactions with environments featuring observation or action delays. Although such methods can be directly trained on the real robots, due to sample inefficiency, limited resources, or safety constraints, a common approach is to transfer models trained in simulation to the physical robot. However, robotic simulations rely on approximated models of the physical systems, which hinders the sim2real transfer. In this work, we consider various uncertainties in modeling the robot or environment dynamics as unknown intrinsic disturbances applied to the system input. We introduce the disturbance-augmented Markov decision process (DAMDP) in delayed settings as a novel representation to incorporate disturbance estimation in training on-policy reinforcement learning algorithms. The proposed method is validated across several metrics on learning robotic reaching and pushing tasks and compared with disturbance-unaware baselines. The results show that the disturbance-augmented models can achieve higher stabilization and robustness in the control response, which in turn improves the prospects of successful sim2real transfer.

Figures (7)

• Figure 1: Delay-resolved RL utilizes action augmentations (modeled as DMDP) to preserve the Markov property in delayed environment settings. Disturbance-aware RL extends the concept to delayed and disturbed environments by exploiting disturbance-augmented observations, where a sequence of estimated disturbances is augmented to the agent observations (modeled as DAMDP) to improve the robustness of the learned control policy.
• Figure 2: Classes of delays imposed on actions, observations, and rewards: (a) no delay, (b) delayed observation with immediate reward, (c) delayed action equivalent to delayed observation with a delayed reward.
• Figure 3: (a) Illustration of how the inverse model is trained with data of a non-randomized simulation. (b) Under the assumption of known delay, DOB employs the trained inverse model to mitigate uncertainties solely via feedforwarding the latest estimated disturbance. (c) Disturbance-aware RL agent uses the delay-resolved DOB with the trained inverse model to achieve control robustness against simulated disturbances in stochastically delayed environments.
• Figure 4: (Left: Simulation, Right: Real) images of the environments with the Kuka LBR iiwa 14 for (a) the reaching task to a green target with the robot's two first joints controlled, and (b) the box pushing task to a red placement target with task space planar control of the pushing pin. In pushing, the box's center of mass lies at the green section's midpoint, showing the purple section is lighter. To estimate the box pose, an ArUco board garrido2014automatic has been used.
• Figure 5: Impact of immediate rewarding vs. delayed rewarding on the reaching experiment for the three distinct representations and the entire average over them. Results are averaged across five random seeds and three randomization sets $\Omega_1 \cup \Omega_2 \cup \Omega_3$. Shaded areas denote the standard error of the mean.

Theorems & Definitions (1)

• Definition 1: DAMDP