Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions

TL;DR

Interleave-VLA tackles the shortcoming of text-only robotic instructions by enabling interleaved image-text inputs, yielding substantial generalization gains in both simulation and real-world manipulation. The approach adds minimal adapter-style changes to existing VLA models, trains on a newly built Open Interleaved X-Embodiment Dataset of 210k episodes, and demonstrates strong zero-shot capabilities across diverse input types (cropped images, web photos, sketches). Empirical results show roughly 2x improvements in out-of-domain generalization and successful cross-embodiment transfer to new robots and backbones, with ablations highlighting the importance of multimodal diversity and instruction imagery. The work also discusses mitigating attentional hallucinations through grounded interleaved signals and provides open data and model-agnostic methodology to scale multimodal robotic learning.

Abstract

The rise of foundation models paves the way for generalist robot policies in the physical world. Existing methods relying on text-only instructions often struggle to generalize to unseen scenarios. We argue that interleaved image-text inputs offer richer and less biased context and enable robots to better handle unseen tasks with more versatile human-robot interaction. Building on this insight, Interleave-VLA, the first robot learning paradigm capable of comprehending interleaved image-text instructions and directly generating continuous action sequences in the physical world, is introduced. It offers a natural, flexible, and model-agnostic paradigm that extends state-of-the-art vision-language-action (VLA) models with minimal modifications while achieving strong zero-shot generalization. Interleave-VLA also includes an automatic pipeline that converts text instructions from Open X-Embodiment into interleaved image-text instructions, resulting in a large-scale real-world interleaved embodied dataset with 210k episodes. Comprehensive evaluation in simulation and the real world shows that Interleave-VLA offers two major benefits: (1) improves out-of-domain generalization to unseen objects by 2x compared to text input baselines, (2) supports flexible task interfaces and diverse instructions in a zero-shot manner, such as hand-drawn sketches. We attribute Interleave-VLA's strong zero-shot capability to the use of instruction images, which effectively mitigate hallucinations, and the inclusion of heterogeneous multimodal datasets, enriched with Internet-sourced images, offering potential for scalability. More information is available at https://interleave-vla.github.io/Interleave-VLA-Anonymous/

Figures (9)

• Figure 1: (a) Our Interleaved X-Embodiment Dataset features diverse, high-quality object-centric images automatically generated from real-world robot demonstrations. (b) Interleave-VLA achieves 2$\times$ stronger out-of-domain generalization compared to text-only VLA models in both simulation and real-robot experiments. (c) It enables flexible, zero-shot instruction following with cropped images, web photos, and hand-drawn sketches for practical and intuitive human-robot interaction.
• Figure 2: Overview of the Interleave-VLA paradigm, featuring an extendable adaptation of Text-VLA to handle interleaved inputs, scalable training on a constructed large interleaved dataset, and versatile inference that supports a wide range of interleaved instructions.
• Figure 3: Left: Our open interleaved X-Embodiment dataset features a large number of high-quality cropped images with diversity across objects. Right: Interleave dataset generation pipeline: (1) Instruction Parsing: use LLM to extract key objects from language instructions. (2) Open-Vocabulary Detection: use OWLv2 to locate and crop target objects from trajectory frames based on the parsed instruction keywords. (3) Data Quality Verification: use Qwen2.5-VL to verify the detected objects, and if needed, provide keypoints for more precise segmentation using Segment Anything.
• Figure 4: Left: Illustration of generalization settings in SIMPLER. (a) Visual generalization: unseen environments, tablecloths, and lighting conditions. (b) Semantic generalization with novel objects from known categories. (c) Semantic generalization with objects from entirely new categories not seen during training. Right: Real-world generalization experiments. In-Domain and out-of-Domain settings in the real world on a FANUC LRMate 200iD/7L robotic arm.
• Figure 5: Qualitative analysis of Interleave-VLA's improved performance over the Text-VLA paradigm. In out-of-domain SimplerEnv tasks with unfamiliar objects, Text-VLA displays attentional hallucination, which typically manifests in three categories: (1)Attention Leakage: the target is partially attended, but focus spills onto irrelevant background or distractor regions; (2)Diffused Attention: attention is broadly scattered with no dominant focus, indicating uncertainty about the target; (3)Attentional Bias: attention centers on a salient distractor instead of the true target. Interleave-VLA effectively mitigates these issues by leveraging in-context visual cues from interleaved instructions, demonstrating consistent attention on target objects.