Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation

TL;DR

Vision-Language-Action models typically rely on visual cues and struggle with contact-rich manipulation due to the absence of tactile feedback. DreamTacVLA addresses this by introducing a three-scale perception pipeline, Hierarchical Spatial Alignment, and a tactile world model that dreams future tactile states within a Think-Dream-Act loop. A pre-trained tactile encoder remains frozen while a lightweight forecasting module predicts future tactile embeddings, enabling anticipatory contact reasoning without expensive planning. Across four contact-rich tasks in simulation and real hardware, the approach achieves up to 95% success and demonstrates the value of tactile grounding and future-state forecasting for robust touch-aware robotics.

Abstract

Vision-Language-Action (VLA) models have shown remarkable generalization by mapping web-scale knowledge to robotic control, yet they remain blind to physical contact. Consequently, they struggle with contact-rich manipulation tasks that require reasoning about force, texture, and slip. While some approaches incorporate low-dimensional tactile signals, they fail to capture the high-resolution dynamics essential for such interactions. To address this limitation, we introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future. Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs coupled with wrist-camera local vision and third-person macro vision. To reconcile these multi-scale sensory streams, we first train a unified policy with a Hierarchical Spatial Alignment (HSA) loss that aligns tactile tokens with their spatial counterparts in the wrist and third-person views. To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals. To mitigate tactile data scarcity and the wear-prone nature of tactile sensors, we construct a hybrid large-scale dataset sourced from both high-fidelity digital twin and real-world experiments. By anticipating upcoming tactile states, DreamTacVLA acquires a rich model of contact physics and conditions its actions on both real observations and imagined consequences. Across contact-rich manipulation tasks, it outperforms state-of-the-art VLA baselines, achieving up to 95% success, highlighting the importance of understanding physical contact for robust, touch-aware robotic agents.

Figures (8)

• Figure 1: Hybrid tactile dataset and the Tactile-DreamVLA inference mechanism. (Top) We collect a large-scale tactile dataset covering 4 manipulation tasks and 9 objects, totaling 2M tactile frames. (Bottom) Our Think–Dream–Act loop executes each step of the policy in two passes. In the Think stage, the policy proposes a draft action using the current state and a null tactile prediction. In the Dream stage, a frozen V-JEPA2 world model forecasts the tactile outcome of that draft action. In the Act stage, the policy integrates both the real observation and the predicted tactile feedback to refine the action. This enables fine-grained corrections for contact-rich manipulation.
• Figure 2: The proposed framework operates in two stages. Stage 1 (Left): A multimodal encoder $E_{\psi}$ processes diverse inputs. This stage employs Hierarchical Spatial Alignment (HSA) to effectively fuse the features from different modalities, guided by the $\mathcal{L}_{HSA}$ and $\mathcal{L}_W$ losses. A policy $\pi_{\theta}$ is trained to output an initial draft action $a^{(t)}_{\text{draft}}$. Stage 2 (Right): A world model $W_{\phi}$ is trained to predict future tactile image sequences. The policy "dreams" the future tactile feeling (e.g., $H^{(t+N)}_{\text{dream}}$) that would result from its draft action. This predicted future is fed into an MLP, allowing the policy to refine its plan and output a more robust final action $a^{(t)}_{\text{final}}$.
• Figure 3: The three-scale visual hierarchy of our model. Our framework fuses information from three distinct visual modalities. Our Hierarchical Spatial Alignment (HSA) loss is designed to explicitly ground the micro-vision (what the robot feels) within the local and macro visual contexts (what the robot sees).
• Figure 4: Visualization of the world model’s predicted future-state embedding $H_{\text{dream}}$ across training. Initially, the embedding is noisy and unstructured, indicating weak predictive ability. As training advances, the embedding becomes increasingly concentrated and stable, revealing that the world model is learning a coherent representation of future tactile–visual dynamics.
• Figure 5: Task suite used to evaluate DreamTacVLA. From left to right: Peg-in-Hole, USB Insert, Gear Assembly, and Tool Stabilization. Each task demands precise, contact-rich manipulation, including aligning tight tolerances, detecting slip, or maintaining stable tool contact. It provides a comprehensive benchmark for assessing tactile-aware policies.