Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception

TL;DR

UniFuture presents a unified driving world model that simultaneously generates future scenes and performs depth-aware perception. It introduces Dual-Latent Sharing to place image and depth in a common latent space and Multi-scale Latent Interaction to enable bidirectional, multi-scale refinement between modalities. Empirical results on nuScenes show superior generation quality (lower FID) and depth perception accuracy, with strong zero-shot generalization to Waymo, highlighting the value of integrating appearance priors with geometric priors. The approach enables coherent 4D scene modeling and points to future work incorporating semantic reasoning for richer scene understanding.

Abstract

We present UniFuture, a simple yet effective driving world model that seamlessly integrates future scene generation and perception within a single framework. Unlike existing models focusing solely on pixel-level future prediction or geometric reasoning, our approach jointly models future appearance (i.e., RGB image) and geometry (i.e., depth), ensuring coherent predictions. Specifically, during the training, we first introduce a Dual-Latent Sharing scheme, which transfers image and depth sequence in a shared latent space, allowing both modalities to benefit from shared feature learning. Additionally, we propose a Multi-scale Latent Interaction mechanism, which facilitates bidirectional refinement between image and depth features at multiple spatial scales, effectively enhancing geometry consistency and perceptual alignment. During testing, our UniFuture can easily predict high-consistency future image-depth pairs by only using the current image as input. Extensive experiments on the nuScenes dataset demonstrate that UniFuture outperforms specialized models on future generation and perception tasks, highlighting the advantages of a unified, structurally-aware world model. The project page is at https://github.com/dk-liang/UniFuture.

Figures (9)

• Figure 1: (a) Popular driving world models focus only on low-level visual representations (i.e., RGB images). (b) Depth estimation models provide high-level geometric information but lack future scene evolution. (c) Our UniFuture unifies future generation and perception in a simple yet effective manner.
• Figure 2: The training pipeline of UniFuture. The Dual-Latent Sharing (DLS) scheme unifies image and depth sequence in a shared latent space without additional pre-training. The image latent undergoes a denoising process, while the depth latent is explicitly predicted. The Multi-scale Latent Interaction (MLI) mechanism enables bidirectional refinement, enhancing consistency in future prediction.
• Figure 3: The inference pipeline of UniFuture, which takes a single image as input and predicts future image-depth pairs. The encoded image latent is concatenated with noise and denoised through the video UNet, followed by a shared decoder that predicts both future images and depth map sequences.
• Figure 4: The details of our proposed Multi-scale Latent Interaction (MLI) mechanism.
• Figure 5: Qualitative comparisons between different models. The existing world model (Vista gao2024vista) lacks depth awareness, leading to inaccurate scene understanding. Depth estimation models (Marigold ke2024repurposing) fail to predict future depth, limiting their ability to capture scene evolution. In contrast, our UniFuture delivers more coherent future predictions, enhancing both generation and perception tasks.