BagelVLA: Enhancing Long-Horizon Manipulation via Interleaved Vision-Language-Action Generation

TL;DR

BagelVLA addresses the challenge of long-horizon manipulation by unifying linguistic planning, visual forecasting, and action generation in a single transformer-based framework. It interleaves the generation of subtask plans $l_t$, future visuals $v_{t+k}$, and actions $a_t$ via a Mixture-of-Transformers and optimizes the joint objective $\mathcal{J}= - (\mathcal{L}_{l}+\mathcal{L}_{v}+\mathcal{L}_{a})$, enabling coherent reasoning and control. A Residual Flow Guidance (RFG) mechanism conditions on the current observation to produce low-latency predictive features, while a two-stage training strategy transfers generic multimodal reasoning to embodied manipulation. Experiments in simulation and real-world benchmarks show BagelVLA outperforming baselines on multi-stage tasks and generalizing to unseen instructions, with notable gains in long-horizon planning and robustness. The approach delivers practical, real-time capable manipulation with strong interpretability through explicit intermediate plans and forecasts.

Abstract

Equipping embodied agents with the ability to reason about tasks, foresee physical outcomes, and generate precise actions is essential for general-purpose manipulation. While recent Vision-Language-Action (VLA) models have leveraged pre-trained foundation models, they typically focus on either linguistic planning or visual forecasting in isolation. These methods rarely integrate both capabilities simultaneously to guide action generation, leading to suboptimal performance in complex, long-horizon manipulation tasks. To bridge this gap, we propose BagelVLA, a unified model that integrates linguistic planning, visual forecasting, and action generation within a single framework. Initialized from a pretrained unified understanding and generative model, BagelVLA is trained to interleave textual reasoning and visual prediction directly into the action execution loop. To efficiently couple these modalities, we introduce Residual Flow Guidance (RFG), which initializes from current observation and leverages single-step denoising to extract predictive visual features, guiding action generation with minimal latency. Extensive experiments demonstrate that BagelVLA outperforms existing baselines by a significant margin on multiple simulated and real-world benchmarks, particularly in tasks requiring multi-stage reasoning.

Figures (13)

• Figure 1: Overview of our framework. We present BagelVLA, a unified model that integrates linguistic planning, visual forecasting, and action generation within a single framework. We construct a massive hybrid dataset combining general multimodal data with large-scale robotic datasets. Robotic datasets with sub-tasks and keyframes are annotated to transfer the foundation model's general reasoning and visual generation abilities to embodied settings.
• Figure 2: Illustration of the BagelVLA framework. BagelVLA utilizes a Mixture-of-Transformers (MoT) architecture, comprising three independent transformers specialized for linguistic, visual, and action modalities. To tackle long-horizon tasks and semantic generalization, we formulate language-conditioned action learning as a long-sequence interleaved planning problem. As shown, we structure these modalities into a unified sequence, enabling the model to generate predictions across all three modalities based on the interleaved context. To support this architecture, we have designed specific mechanisms to facilitate interaction among multiple flow-matching experts and to enhance inference efficiency.
• Figure 3: Illustration of different types of dual denoising schemes. (a) Complete Denoise: Image prediction and action generation are performed separately, requiring a total of $N_1+N_2$ denoising steps. (b) Joint Denoise: Image prediction and action generation are performed simultaneously, denoising together for $N$ steps. (c) Single-Step Denoise: Action generation is conditioned directly on the context from the first denoising step of the image prediction. Further implementation details, including the construction of the concatenated sequence and the attention mask are provided in Appendix \ref{['sec:app-dualfm']}.
• Figure 4: Visualization of interleaved planning results on real-world robotic tasks. Given a global instruction and the current observation, BagelVLA leverages the context to identify the immediate subtask, predicts a goal image for that subtask, and subsequently generates actions. The figure illustrates the interleaved planning results for Stack Cubes in Requested Order, Calculate and Place Symbol Blocks task, and a task from the Agibot dataset agibotworldcontributors2025agibotworldcolosseolargescale. More cases can be found in Appendix \ref{['sec:app-inter']}.
• Figure 5: Predicted images using different denoising steps. The figure displays the generation results for the naive single-step denoise (Eq. \ref{['vpp1']}) and RFG (Eq. \ref{['vpp2']}) across varying denoising steps in real-world basic tasks and Robotwin randomized (unseen) scenarios. RFG demonstrates the capability to preserve backgrounds and achieve high-quality generation with very few steps. This provides strong support for reducing the inference latency of interleaved generation. More cases can be found in Appendix \ref{['sec:app-rfg']}.