WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning

Abstract

Autonomous driving systems depend on on models that can reason about high-level scene contexts and accurately predict the dynamics of their surrounding environment. Vision- Language Models (VLMs) have recently emerged as promising tools for decision-making and scene understanding, offering strong capabilities in contextual reasoning. However, their limited spatial comprehension constrains their effectiveness as end-to-end driving models. World Models (WM) internalize environmental dynamics to predict future scene evolution. Recently explored as ego-motion predictors and foundation models for autonomous driving, they represent a promising direction for addressing key challenges in the field, particularly enhancing generalization while maintaining dynamic prediction. To leverage the complementary strengths of context-based decision making and prediction, we propose WorldVLM: A hybrid architecture that unifies VLMs and WMs. In our design, the high-level VLM generates behavior commands to guide the driving WM, enabling interpretable and context-aware actions. We evaluate conditioning strategies and provide insights into the hybrid design challenges.

Figures (6)

• Figure 1: We propose WorldVLM, a hybrid framework combining Vision-Language based Reasoning for high-level behavior planning and World Model forecasting for ego-trajectory prediction.
• Figure 2: WorldVLM Framework: The VLM receives front images alongside a textual prompt to generate a scene-based justification and action description and generate structured behavior commands. These commands supervise a latent driving WM li2025enhancing that takes visual latent features encoded from surrounding images to predict future visual latent features, extracting the ego-trajectory from the scene dynamics.
• Figure 3: Behavior conditioning of the LAW li2025enhancing model. The behavior is concatenated with waypoint queries and spatial view features into the Waypoint Transformer Decoder and concatenated for WM prediction.
• Figure 4: Sample of our Justification and Action Description dataset.
• Figure 5: Justification and Action Description generated by the behavior-planning VLM following our dataset annotation scheme.