GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation

TL;DR

GaussianDWM tackles the lack of explicit 3D–language alignment in driving world models by introducing a 3D Gaussian scene representation where each $G_i=(x_i,o_i,s_i,r_i,f_i)$ embeds language features. It uses a World Tokenizer, a Language-augmented Scene Understanding module, and a dual-condition diffusion-based Multi-modal Generator to perform spatial and temporal scene generation guided by high-level world knowledge and low-level visual cues. The model achieves state-of-the-art performance on NuScenes and NuInteract for both scene understanding and generation, with ablations confirming the effectiveness of 3D Gaussians, task-aware sampling, and world-knowledge conditioning. This work enables more interpretable, semantically coherent driving scene reasoning and synthesis, with practical implications for simulation, risk assessment, and planning in autonomous driving.

Abstract

Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub https://github.com/dtc111111/GaussianDWM.

Figures (8)

• Figure 1: We propose the first unified 3D Gaussian-based world model framework that achieves comprehensive scene understanding and scene generation for driving scenarios. It efficiently encodes complex scenes, samples task-relevant information, and handles diverse question-answering tasks. Moreover, by leveraging the extracted world knowledge, our framework guides the generative model to perform accurate spatial and temporal scene generation.
• Figure 2: System Overview. We propose the first unified 3D Gaussian-based world model framework that simultaneously supports both scene understanding and scene generation. We first employ a scene encoder to align the language information with the 3D Gaussians, resulting in language-augmented 3D Gaussian representations. Then, a designed Gaussian projector aligns the 3D Gaussian tokens, 2D image tokens, and text tokens into a unified latent space. Subsequently, a task-aware hybrid sampling strategy is applied to select the most relevant 3D Gaussian tokens for the current query, which are then fed into the LLM. The LLM produces both textual answers and high-level language features that encapsulate world knowledge, which are later used to guide multi-modal scene generation.
• Figure 3: Qualitative results for scene understanding and scene generation. From top to bottom, we display the multi-view input of the current scene and the 3D Gaussian ellipsoids, the scene understanding results, and the spatial and temporal scene generation results.
• Figure 4: Qualitative comparison of RGB-D NVS with 2m shift. Compared with state-of-the-art reconstruction-based methods for spatial NVS omniredeformable3dgspvgstreetgaussian, our method reduce artifacts of dynamic objects and preserves temporal-spatial consistency across large viewpoint shifts.
• Figure 5: World-Knowledge Ablation. We visualize the effect of world knowledge from LLM under a 4m left-shifted novel view. From left to right: ground-truth images at the original viewpoints (CAM_FRONT and CAM_FRONT_LEFT), world knowledge predicted by our GaussianDWM, novel-view synthesis with world knowledge, and novel-view synthesis without world knowledge. Regions where world knowledge improves semantic and geometric fidelity are highlighted with bounding boxes and zoomed-in at the center. The results show that incorporating world knowledge enables the model to better understand scene context and generate more realistic, semantically consistent geometry.