FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation

TL;DR

FD-VLA tackles the challenge of integrating force information into vision-language-action models for contact-rich manipulation without relying on physical force sensors. It introduces the Force Distillation Module (FDM), which learns a latent force token from vision and robot state during training and injects this token into a frozen Vision-Language Model via a directional masking scheme, enabling force-aware reasoning while preserving pretrained semantic alignment. The approach is trained with a force-distillation loss in addition to a flow-matching policy objective, and an action expert decodes maneuver sequences from fused multimodal features. Real-world experiments on a UR5e platform show that FD-VLA outperforms strong baselines and that the learned latent force representations provide robustness and sensor-free deployment across diverse contact-rich tasks.

Abstract

Force sensing is a crucial modality for Vision-Language-Action (VLA) frameworks, as it enables fine-grained perception and dexterous manipulation in contact-rich tasks. We present Force-Distilled VLA (FD-VLA), a novel framework that integrates force awareness into contact-rich manipulation without relying on physical force sensors. The core of our approach is a Force Distillation Module (FDM), which distills force by mapping a learnable query token, conditioned on visual observations and robot states, into a predicted force token aligned with the latent representation of actual force signals. During inference, this distilled force token is injected into the pretrained VLM, enabling force-aware reasoning while preserving the integrity of its vision-language semantics. This design provides two key benefits: first, it allows practical deployment across a wide range of robots that lack expensive or fragile force-torque sensors, thereby reducing hardware cost and complexity; second, the FDM introduces an additional force-vision-state fusion prior to the VLM, which improves cross-modal alignment and enhances perception-action robustness in contact-rich scenarios. Surprisingly, our physical experiments show that the distilled force token outperforms direct sensor force measurements as well as other baselines, which highlights the effectiveness of this force-distilled VLA approach.

Figures (7)

• Figure 1: Overview of differentiate architectures of force VLAs. (Left) Tactile-VLA with tactile encoder directly encode tactile information. (Middle) Force-VLA with MoE module between VLM and action expert. (Right) Our FD-VLA using predicted force instead of actual force to get better fused feature and sensor free at deployment.
• Figure 2: Overview of our framework. During training, measured force signals are encoded into an actual force token via a lightweight projection. A learnable query attends to image and state tokens to predict a latent force token, which is supervised against the actual force token. At inference, the model synthesizes this latent force representations solely from vision and state inputs, eliminating the need for tactile hardware. The predicted force token is then fused with language, image, and state tokens inside the pretrained VLM, and an action expert consumes the fused representation to generate action sequences.
• Figure 3: Visualization of raw force in the plug insertion task.
• Figure 4: Visualization of real-world experimental tasks: 1) Clean the whiteboard, 2) Press the emergency button, 3) Insert the plug into the socket.
• Figure 5: Visualization of the real robotic platform. We use a UR5e robot arm as the main manipulation platform, the Kinect Azure camera as the main camera and the RealSense D405 as the gripper camera.