Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wrist

TL;DR

The paper addresses robust manipulation under uncertainty by combining tactile memory with a soft-wrist robot to safely collect data and adapt to unseen scenarios. It introduces MAT3, a Masked Tactile Trajectory Transformer that jointly encodes spatiotemporal interactions across distributed taxels and auxiliary modalities via masked token prediction, producing rich representations stored in tactile memory. A non-parametric retrieval policy uses nearest-neighbor search over memory embeddings to replay actions, enabling adaptation without explicit subtask segmentation. Real-robot peg-in-hole experiments across seven peg shapes and various perturbations show MAT3 achieves higher success rates than baselines, including under unseen conditions, highlighting the practical potential of tactile memory integrated with soft, compliant sensing for contact-rich manipulation.

Abstract

Tactile memory, the ability to store and retrieve touch-based experience, is critical for contact-rich tasks such as key insertion under uncertainty. To replicate this capability, we introduce Tactile Memory with Soft Robot (TaMeSo-bot), a system that integrates a soft wrist with tactile retrieval-based control to enable safe and robust manipulation. The soft wrist allows safe contact exploration during data collection, while tactile memory reuses past demonstrations via retrieval for flexible adaptation to unseen scenarios. The core of this system is the Masked Tactile Trajectory Transformer (MAT$^\text{3}$), which jointly models spatiotemporal interactions between robot actions, distributed tactile feedback, force-torque measurements, and proprioceptive signals. Through masked-token prediction, MAT$^\text{3}$ learns rich spatiotemporal representations by inferring missing sensory information from context, autonomously extracting task-relevant features without explicit subtask segmentation. We validate our approach on peg-in-hole tasks with diverse pegs and conditions in real-robot experiments. Our extensive evaluation demonstrates that MAT$^\text{3}$ achieves higher success rates than the baselines over all conditions and shows remarkable capability to adapt to unseen pegs and conditions.

Figures (6)

• Figure 1: TaMeSo-bot uses softness (i.e., mechanical compliance) to safely collect tactile demonstrations (left) and flexibly adapt to unseen conditions (right) through retrieval from tactile memory.
• Figure 2: Overview of the TaMeSo-bot system. The tactile memory system stores and retrieves tactile information from a database of encoded demonstrations, enabling robust peg-in-hole manipulation by matching current sensory inputs to similar past experiences.
• Figure 3: Masked Tactile Trajectory Transformer (MAT$^\text{3}$) for distributed taxels. We integrate auxiliary and spatiotemporal information as soft and hard-concatenation of the embeddings. During training, the encoder learns to reconstruct the states and actions within a time window $H$, while randomly masking input tokens. After training, the tactile trajectory datasets are encoded into a single representative embedding $\bf z$ that captures the spatio-temporal dynamics of the tactile-action sequences.
• Figure 4: Seen and unseen peg and dimension (mm)
• Figure 5: Unseen conditions in the experiments.