Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation

TL;DR

This work addresses the force-blindness of vision-language-action policies in robotic manipulation by introducing tactile-force alignment. It presents TaF-VLA, which pairs a large-scale TaF-Dataset with the TaF-Adapter to ground tactile observations in physical forces using contrastive learning and vector quantization, then integrates this into a VLA backbone for force-aware control. Across seven force-critical tasks, TaF-VLA substantially outperforms vision-only and tactile-vision baselines, demonstrating improved robustness, data efficiency, and cross-sensor generalization. The approach enables language-conditioned manipulation that modulates force based on tactile feedback, highlighting the practical impact of explicitly grounding tactile perception in physical dynamics for dexterous, generalist manipulation.

Abstract

Vision-Language-Action (VLA) models have recently emerged as powerful generalists for robotic manipulation. However, due to their predominant reliance on visual modalities, they fundamentally lack the physical intuition required for contact-rich tasks that require precise force regulation and physical reasoning. Existing attempts to incorporate vision-based tactile sensing into VLA models typically treat tactile inputs as auxiliary visual textures, thereby overlooking the underlying correlation between surface deformation and interaction dynamics. To bridge this gap, we propose a paradigm shift from tactile-vision alignment to tactile-force alignment. Here, we introduce TaF-VLA, a framework that explicitly grounds high-dimensional tactile observations in physical interaction forces. To facilitate this, we develop an automated tactile-force data acquisition device and curate the TaF-Dataset, comprising over 10 million synchronized tactile observations, 6-axis force/torque, and matrix force map. To align sequential tactile observations with interaction forces, the central component of our approach is the Tactile-Force Adapter (TaF-Adapter), a tactile sensor encoder that extracts discretized latent information for encoding tactile observations. This mechanism ensures that the learned representations capture history-dependent, noise-insensitive physical dynamics rather than static visual textures. Finally, we integrate this force-aligned encoder into a VLA backbone. Extensive real-world experiments demonstrate that TaF-VLA policy significantly outperforms state-of-the-art tactile-vision-aligned and vision-only baselines on contact-rich tasks, verifying its ability to achieve robust, force-aware manipulation through cross-modal physical reasoning.

Figures (10)

• Figure 1: From Data to Policy: The TaF-VLA Pipeline. To address the "force-blindness" of current vla models, we propose a paradigm shift from tactile-vision to tactile-force alignment, realized through three stages: (a) We deploy an automated data acquisition system (TaF-Device) to construct the TaF-Dataset, a large-scale collection of synchronized visuotactile images, 6-axis force/torque, and matrix force maps. Using this data, we pretrain the TaF-Adapter to align tactile observations with ground-truth force signals in a shared latent space. (b) We fuse the TaF-Adapter into a VLA backbone and fine-tune the policy on real-world demonstrations enriched with force-aware language instructions (i.e., force-aware manipulation dataset). (c) This explicit tactile-force alignment empowers TaF-VLA to master complex force-aware manipulation tasks, such as tool-use and deformable object manipulation, where traditional vision-based baselines consistently fail.
• Figure 2: The hardware design of the tactile-force data acquisition device that autonomously captures synchronized paired frames. (a) Overall view of the device. (b) Kinematics of the platforms. (c) Force and tactile sensors that mounted on the system. In particular, the vision-based tactile sensor includes four self-made sensors (each silicone surface carrying a distinct marker pattern) and two GelSight Mini supplied in both marker-printed and marker-free versions. (d) Example contact indenter samples with varying patterns, curvatures, and hardness. (e) Synchronized tactile-force frames.
• Figure 3: Overview of the TaF-VLA learning framework. The architecture consists of two distinct stages: (a) Tactile-Force Alignment (TaF-Adapter): We propose a self-supervised learning paradigm. The Force Encoder ($f_{\phi'}, f_{\phi"}$) quantizes heterogeneous force signals, distributed pressure maps, and 6-axis F/T vectors, into two discrete codebooks via vqvae, creating stable physical anchors. The Tactile Encoder ($f_\textrm{causal-TF}$) employs a Causal Transformer to process sequential tactile images. These branches are synchronized in a shared latent space via a contrastive InfoNCE objective ($\mathcal{L}_\text{NCE}$), effectively teaching the tactile encoder to infer force dynamics from visual deformation. (b) Policy Integration (TaF-VLA): The pre-trained, frozen TaF-Adapter is integrated into a vla backbone. It injects force-aligned tactile tokens alongside visual and language embeddings, allowing the Action Expert to generate force-aware manipulation trajectories.
• Figure 4: Overview of the eight force-aware manipulation tasks. Blue arrows indicate end-effector trajectories.
• Figure 5: Successful execution sequences across the evaluated task suite: (a) Tube Insertion; (b) Tweezer Weight Pick; (c) Power Bank Extraction; (d) Whiteboard Erasing; (e) Jelly Slicing; (f1) Heavy Part Lifting (Buckle); (f2) Heavy Part Lifting (Door Handle); (h) Tongs Food Pick; (i) Chocolate Grasping.