HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

TL;DR

Hermes presents a unified Driving World Model that jointly performs 3D scene understanding and future scene generation by encoding multi-view observations into a BEV latent space and enriching an LLM with world knowledge via world queries. The framework uses a BEV-based tokenizer and a differentiable BEV-to-point renderer, connected through a current-to-future link that fuses current BEV features with questions and ego-motion to predict future point clouds. It achieves state-of-the-art generation performance (32.4% lower Chamfer distance) and strong understanding metrics (8.0% CIDEr gains) on nuScenes and OmniDrive-nuScenes, while enabling scene descriptions and reasoning. The work demonstrates effective cross-task knowledge transfer and signals a practical step toward a fully unified self-driving world model, with future directions including perception tasks and alternative generation modalities.

Abstract

Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which involves interpreting and reasoning about the driving environment. In this paper, we present a unified Driving World Model named HERMES. We seamlessly integrate 3D scene understanding and future scene evolution (generation) through a unified framework in driving scenarios. Specifically, HERMES leverages a Bird's-Eye View (BEV) representation to consolidate multi-view spatial information while preserving geometric relationships and interactions. We also introduce world queries, which incorporate world knowledge into BEV features via causal attention in the Large Language Model, enabling contextual enrichment for understanding and generation tasks. We conduct comprehensive studies on nuScenes and OmniDrive-nuScenes datasets to validate the effectiveness of our method. HERMES achieves state-of-the-art performance, reducing generation error by 32.4% and improving understanding metrics such as CIDEr by 8.0%. The model and code will be publicly released at https://github.com/LMD0311/HERMES.

Figures (8)

• Figure 1: (a) Previous driving world models focus on generative scene evolution prediction. (b) Large language models for driving are limited to scene understanding. (c) A straightforward unification manner using the generator and large language models separately with a shared feature. (d) The proposed simple framework unifies 3D scene understanding and generates scene evolution based on given actions.
• Figure 2: The pipeline of our Hermes. The BEV tokenizer converts multi-view $I_{t}$ into flattened BEV $\boldsymbol{\mathcal{F}}_{t}$, which are fed into the large language model (LLM). The LLM interprets user instructions $\boldsymbol{\mathcal{T}}$ and generates textual responses by leveraging its understanding of driving scenes as world knowledge. A group of world queries $\boldsymbol{\mathcal{Q}}^{w}$ are appended to the LLM input sequence. Encoded BEV $\boldsymbol{\mathcal{B}}_{t}$ and world queries generate future BEV ($\boldsymbol{\mathcal{B}}_{t+1},\cdots,\boldsymbol{\mathcal{B}}_{t+\Delta t}$) via a current to future link, and the shared Render generates point clouds evolution.
• Figure 3: The effect of world queries for understanding (CIDEr) and generation (3s chamfer distance) is trained on full data.
• Figure 4: Qualitative results for future generation and scene understanding. From top to bottom, we display the multi-view input of the current scene, the ground truth scene evolution, the generated scene evolution, and the scene understanding result.
• Figure S1: Qualitative results of Hermes conditioned on different future ego-motion conditions. From top to bottom, each sub-figure displays the multi-view input of the current scene, scene evolution predicted with a "stop" future ego-motion, and scene evolution predicted with a "turn right" ego-motion.