HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing

Abstract

Tactile sensing is a crucial capability for Vision-Language-Action (VLA) architectures, as it enables dexterous and safe manipulation in contact-rich tasks. However, reliance on dedicated tactile hardware increases cost and reduces reproducibility across robotic platforms. We argue that tactile-aware manipulation can be learned offline and deployed without direct haptic feedback at inference. To this end, we present HapticVLA, which proceeds in two tightly coupled stages: Safety-Aware Reward-Weighted Flow Matching (SA-RWFM) and Tactile Distillation (TD). SA-RWFM trains a flow-matching action expert that incorporates precomputed, safety-aware tactile rewards penalizing excessive grasping force and suboptimal grasping trajectories. TD further transfers this tactile-aware capability into a conventional VLA: we distill a compact tactile token from the SA-RWFM teacher and train a student VLA to predict that token from vision and state modalities, enabling tactile-aware action generation at inference without requiring on-board tactile sensors. This design preserves contact-rich tactile-aware reasoning within VLA while removing the need for on-board tactile sensors during deployment. On real-world experiments, HapticVLA achieves a mean success rate of 86.7%, consistently outperforming baseline VLAs - including versions provided with direct tactile feedback during inference.

Figures (6)

• Figure 1: HapticVLA is the first VLA model to enable tactile-aware manipulation without requiring force sensors on inference. Alongside vision, language, and state modalities it processes tactile reward enabling safe action generation for contact-rich manipulation. Our approach leverages tactile distillation to preserve haptic-aware reasoning while eliminating tactile modality through distillation.
• Figure 2: Framework overview of HapticVLA. After dataset collection we perform offline tactile reward calculation for each episode based on manipulator state and tactile maps. During VLA training, along with language, vision and state modalities passed into VLM, this reward is passed as an additional condition directly into SA-RWFM action expert. To eliminate the need for tactile hardware, we further perform tactile distillation on our trained VLA. At inference, the distilled VLA predicts tactile representations solely from vision and state inputs enabling tactile-aware contact-rich manipulation.
• Figure 3: Our dataset collection setups: (Left) Real robotics platform; and (Right) Digital twin in Isaac Sim.
• Figure 4: (Left) Tactile array with 100 taxels. (Right) Example of 10 x 10 tactile map from the array during manipulation. Red/orange regions indicate high contact forces and blue regions low contact forces.
• Figure 5: For our robotics platform we use two SO-101 robot arms as the main manipulation platform, an Intel Realsense D 435 as the main camera and wrist-mounted IMX335 cameras.