Spatially anchored Tactile Awareness for Robust Dexterous Manipulation

TL;DR

SaTA addresses the challenge of sub-millimeter dexterous manipulation by explicitly grounding tactile signals in the hand's kinematic frame, enabling precise geometric reasoning without object models. It introduces a Spatial Anchored Tactile Encoder that uses forward kinematics to attach tactile measurements to the hand, augments them with Fourier-based multi-scale pose encoding, and fuses them via FiLM, all processed by an ACT-style Transformer. Trained through imitation from expert demonstrations, SaTA significantly outperforms vision-only and non-grounded tactile baselines across bimanual USB-C mating, light bulb threading, and card sliding, with up to a 30 percentage-point increase in success rate and about a 27% reduction in completion time. This work demonstrates that preserving tactile texture while anchoring measurements to a stable hand frame enables direct geometric inference for precise manipulation, establishing a general principle for tactile-driven learning in robotics.

Abstract

Dexterous manipulation requires precise geometric reasoning, yet existing visuo-tactile learning methods struggle with sub-millimeter precision tasks that are routine for traditional model-based approaches. We identify a key limitation: while tactile sensors provide rich contact information, current learning frameworks fail to effectively leverage both the perceptual richness of tactile signals and their spatial relationship with hand kinematics. We believe an ideal tactile representation should explicitly ground contact measurements in a stable reference frame while preserving detailed sensory information, enabling policies to not only detect contact occurrence but also precisely infer object geometry in the hand's coordinate system. We introduce SaTA (Spatially-anchored Tactile Awareness for dexterous manipulation), an end-to-end policy framework that explicitly anchors tactile features to the hand's kinematic frame through forward kinematics, enabling accurate geometric reasoning without requiring object models or explicit pose estimation. Our key insight is that spatially grounded tactile representations allow policies to not only detect contact occurrence but also precisely infer object geometry in the hand's coordinate system. We validate SaTA on challenging dexterous manipulation tasks, including bimanual USB-C mating in free space, a task demanding sub-millimeter alignment precision, as well as light bulb installation requiring precise thread engagement and rotational control, and card sliding that demands delicate force modulation and angular precision. These tasks represent significant challenges for learning-based methods due to their stringent precision requirements. Across multiple benchmarks, SaTA significantly outperforms strong visuo-tactile baselines, improving success rates by up to 30 percentage while reducing task completion times by 27 percentage.

Figures (4)

• Figure 1: The importance of spatial grounding for tactile-based geometric reasoning. Left: Without spatial grounding, tactile sensors detect contact but cannot infer correct alignment—rich tactile patterns remain isolated measurements. Right: With spatial grounding in the hand's kinematic frame, the same tactile signals enable precise geometric reasoning—each measurement's spatial context guides successful USB-C mating.
• Figure 2: SaTA Framework Overview. Our method grounds raw tactile images in the hand's kinematic frame for dexterous manipulation. Multi-modal inputs (head/wrist RGB views, tactile images from fingertips, joint states) are processed through specialized encoders. The core innovation is the Spatial Anchored Tactile Encoder (yellow), which grounds each tactile measurement in the hand's kinematic frame. The detailed SAT Encoding process (gray box) shows how we compute 6D poses via forward kinematics, apply Fourier encoding for multi-scale spatial representation, and use FiLM to integrate spatial context with tactile features from ResNet. All encoded features are processed through an ACT-style Transformer to generate action sequences for precise manipulation.
• Figure 3: Task overview and challenges. Each row represents one task: (top) USB-C mating, (middle) card sliding, (bottom) bulb installation. From left to right, columns show: initial state, approach phase, critical contact moment (red boxes), and successful completion. Red boxes highlight moments where visual information is severely degraded, requiring precise tactile-based geometric reasoning.
• Figure 4: Failure mode analysis of Tactile-Flat baseline. Red arrows indicate incorrect directions/orientations executed by the policy, while green arrows show the required correct directions. (Top) Bulb installation: misaligned insertion angle prevents thread engagement. (Middle) Card sliding: perpendicular thumb motion (red) instead of along-surface motion (green) causes card bending (blue curve). (Bottom) USB-C mating: incorrect plug orientation (red) and absence of thumb-index coordinated adjustment motion (blue arrows indicate the required but unlearned motion).