ProDapt: Proprioceptive Adaptation using Long-term Memory Diffusion

TL;DR

ProDapt tackles the challenge of executing manipulation tasks with only proprioceptive sensing by endowing diffusion-based policies with long-term memory through a curated set of keypoints that capture past contacts. The method conditions the diffusion controller on both recent observations and stored keypoints, enabling robust task completion without exteroceptive data. Through simulation and real UR10e experiments, ProDapt demonstrates that memory-augmented diffusion reduces bias, improves success rates across complex obstacle configurations, and speeds up convergence compared to memoryless baselines. This approach advances robust proprioceptive control for harsh or sensor-failing environments, with potential applications in space and underwater exploration where exteroceptive sensing is unreliable.

Abstract

Diffusion models have revolutionized imitation learning, allowing robots to replicate complex behaviours. However, diffusion often relies on cameras and other exteroceptive sensors to observe the environment and lacks long-term memory. In space, military, and underwater applications, robots must be highly robust to failures in exteroceptive sensors, operating using only proprioceptive information. In this paper, we propose ProDapt, a method of incorporating long-term memory of previous contacts between the robot and the environment in the diffusion process, allowing it to complete tasks using only proprioceptive data. This is achieved by identifying "keypoints", essential past observations maintained as inputs to the policy. We test our approach using a UR10e robotic arm in both simulation and real experiments and demonstrate the necessity of this long-term memory for task completion.

Figures (8)

• Figure 1: Our proposed approach with the novel components in red. The proprioceptive sensor measurements $\textbf{y}_k$ are provided to the keypoint manager, which determines if the robot has contacted an obstacle and whether that contact is unique. If so, that contact is saved as a keypoint. During inference, the diffusion model is conditioned on the most useful keypoints and previous observations and outputs the next actions.
• Figure 2: Experimental setup on a simulated UR10e in Isaac Sim and a real UR10e. In both setups, the arm must move from a starting position to a goal position despite unknown obstacles and no exteroception.
• Figure 3: Our approach is evaluated on four experiment setups that test long-term memory, reasoning, and performance.
• Figure 4: The percentage of successful trials for each simulated experiment setup for our approach and the baselines.
• Figure 5: The time required to complete each simulated experiment setup for our approach and the baselines. Only successful trials are plotted.