RynnVLA-002: A Unified Vision-Language-Action and World Model

TL;DR

RynnVLA-002 addresses the gap between vision-language-action models and physics-aware world models by unifying them in a single framework $M_\psi$ that can be queried as either a VLA or a world model. It employs three tokenizers with a shared vocabulary, discrete action tokens, and a parallel continuous Action Transformer, using an action attention masking strategy and a combined loss $\mathcal{L} = \mathcal{L}_{dis} + \alpha \mathcal{L}_{conti}$ to enable both short-horizon action generation and long-horizon planning. Training on mixed VLA and world-model data, RynnVLA-002 achieves 97.4% success on the LIBERO simulation benchmark without pretraining and boosts real-world LeRobot performance by about 50% when the world model is integrated, outperforming standalone VLA and world models. These results demonstrate mutual enhancement between perception, dynamics prediction, and action generation, offering a practical foundation for unified multimodal embodied AI across text, vision, and action.

Abstract

We introduce RynnVLA-002, a unified Vision-Language-Action (VLA) and world model. The world model leverages action and visual inputs to predict future image states, learning the underlying physics of the environment to refine action generation. Conversely, the VLA model produces subsequent actions from image observations, enhancing visual understanding and supporting the world model's image generation. The unified framework of RynnVLA-002 enables joint learning of environmental dynamics and action planning. Our experiments show that RynnVLA-002 surpasses individual VLA and world models, demonstrating their mutual enhancement. We evaluate RynnVLA-002 in both simulation and real-world robot tasks. RynnVLA-002 achieves 97.4% success rate on the LIBERO simulation benchmark without pretraining, while in real-world LeRobot experiments, its integrated world model boosts the overall success rate by 50%.

Figures (9)

• Figure 1: (a) VLA model generates actions based on image understanding; (b) World model generates the image based on image and action understanding; (c) Action World Model unifies both image and action understanding and generation.
• Figure 2: Overview of RynnVLA-002. RynnVLA-002 involves VLA model data and world model data during the training process.
• Figure 3: Attention mask of (a) default VLA model, (2) our proposed VLA model, and (c) world model.
• Figure 4: Real-world robot settings. (a) Place the block inside the circle. (b) Place the strawberries into the cup. (c) Task with distractors.
• Figure 5: VLA model visualization on LIBERO. Task: put the cream cheese in the bowl. Top: w/o world model. Bottom: w/ world model.