VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

TL;DR

VTLA tackles insertion manipulation by integrating vision, tactile sensing, and language through VGTE tokens and a Direct Preference Optimization training regime. The approach leverages a large simulated visuotactile dataset with domain randomization and a cross-modal instruction tuning strategy to enable robust policy generation. Empirical results show VTLA outperforms diffusion-based and other multi-modal baselines in both simulation and real-world peg-in-hole tasks, with strong Sim2Real transfer and high success rates for unseen peg shapes. The work advances tactile-embedded vision-language-action models and opens avenues for more reliable contact-rich robotic manipulation in unstructured settings.

Abstract

While vision-language models have advanced significantly, their application in language-conditioned robotic manipulation is still underexplored, especially for contact-rich tasks that extend beyond visually dominant pick-and-place scenarios. To bridge this gap, we introduce Vision-Tactile-Language-Action model, a novel framework that enables robust policy generation in contact-intensive scenarios by effectively integrating visual and tactile inputs through cross-modal language grounding. A low-cost, multi-modal dataset has been constructed in a simulation environment, containing vision-tactile-action-instruction pairs specifically designed for the fingertip insertion task. Furthermore, we introduce Direct Preference Optimization (DPO) to offer regression-like supervision for the VTLA model, effectively bridging the gap between classification-based next token prediction loss and continuous robotic tasks. Experimental results show that the VTLA model outperforms traditional imitation learning methods (e.g., diffusion policies) and existing multi-modal baselines (TLA/VLA), achieving over 90% success rates on unseen peg shapes. Finally, we conduct real-world peg-in-hole experiments to demonstrate the exceptional Sim2Real performance of the proposed VTLA model. For supplementary videos and results, please visit our project website: https://sites.google.com/view/vtla

Figures (8)

• Figure 1: Overview of VTLA. The VTLA model learns a robotic manipulation policy integrated with vision, tactile, and language inputs from domain-randomized simulation data, enabling it to perform a variety of peg-in-hole tasks in the real world.
• Figure 2: The data collection diagram and data examples of the VTLA dataset.
• Figure 3: The pipeline of VTLA. In stage 1, the instruction dataset is created from simulation data using a vision-guided temporal enhancement, and the VTLA model is optimized with NTP loss. In stage 2, DPO is introduced to provide regression-like supervision, bridging the gap between VLM training and robotic continuous control, thereby enhancing performance.
• Figure 4: Snapshots of real-world insertion using the proposed VTLA model and baseline methods (VLA and TLA). The initial state of the task is consistent, and the assembly clearance is 0.6 mm.
• Figure 5: The insertion task setup in the real world. The left part shows the robot platform, and the right part shows the real visual-tactile observations and a dialogue round.