Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving

TL;DR

This work addresses environment prediction for autonomous driving under partial observability and uncertainty by forecasting LiDAR-based occupancy grids. It introduces Latent Occupancy Prediction (LOPR), a self-supervised framework that learns a compact latent representation of L-OGMs via a β-VAE–GAN and uses a stochastic autoregressive transformer to model multi-future predictions conditioned on RGB, maps, and planned trajectories. A real-time single-step decoder provides fast predictions, while a diffusion-based batch decoder refines temporal coherence when needed. Across nuScenes and Waymo Open, LOPR outperforms prior OGM methods and latent-space baselines, with additional gains from multimodal conditioning and stochasticity, highlighting its practical potential for AV planning.

Abstract

Environment prediction frameworks are critical for the safe navigation of autonomous vehicles (AVs) in dynamic settings. LiDAR-generated occupancy grid maps (L-OGMs) offer a robust bird's-eye view for the scene representation, enabling self-supervised joint scene predictions while exhibiting resilience to partial observability and perception detection failures. Prior approaches have focused on deterministic L-OGM prediction architectures within the grid cell space. While these methods have seen some success, they frequently produce unrealistic predictions and fail to capture the stochastic nature of the environment. Additionally, they do not effectively integrate additional sensor modalities present in AVs. Our proposed framework, Latent Occupancy Prediction (LOPR), performs stochastic L-OGM prediction in the latent space of a generative architecture and allows for conditioning on RGB cameras, maps, and planned trajectories. We decode predictions using either a single-step decoder, which provides high-quality predictions in real-time, or a diffusion-based batch decoder, which can further refine the decoded frames to address temporal consistency issues and reduce compression losses. Our experiments on the nuScenes and Waymo Open datasets show that all variants of our approach qualitatively and quantitatively outperform prior approaches.

Figures (8)

• Figure 1: Latent Occupancy PRediction (LOPR) is a self-supervised stochastic prediction framework that forecasts occupancy grid maps within the latent space of a generative model. It consists of deterministic and variational transformer modules conditioned on occupancy grids, images, maps, and the planned trajectory. LOPR forecasts multiple plausible futures for the entire scene.
• Figure 2: The LOPR framework, comprising sensor processing, stochastic inference, and prediction modules. The sensor processing module encodes all sensor modalities. The L-OGM and RGB camera encoders are pretrained as described in \ref{['sec:unsupervised']} and in \ref{['sec:modality']}. The inference module captures the scene's stochasticity (\ref{['sec:prediction']}). In the prediction module, we forecast the next time step's L-OGM embedding. At each time step, the most recent predictions are autoregressively provided to the inference and prediction modules.
• Figure 3: Vision Transformer-based RGB Camera Encoder. RGB camera data is processed through the pre-trained DINOv2 backbone, subsequently passing through a series of attention layers. These layers aggregate information within each view (image layer), across different views (spatial layer), and throughout all observed timesteps (temporal layer). The spatial and temporal layers also include the learned positional embedding and are conditioned on the L-OGM embeddings and the planned trajectory, respectively.
• Figure 4: Our encoder-decoder effectively reconstructs the L-OGMs, with the IS score reflecting differences in the distribution of occupied cells (e.g., IS of 2.13 and 3.69). In rare instances (e.g., rotation with many agents, IS=5.04), it might lose some detail. White represents occupied cells, black denotes free space, and shades of gray capture values between 0 and 1.
• Figure 5: Examples of LOPR and OccWorld-2D predictions with visualized front camera observations from the nuScenes dataset. The predictions are generated with the single-step decoder. LOPR is conditioned on all cameras around the vehicle, maps, and the planned trajectory. We report IS scores for each sample. (Left) Prediction of an oncoming vehicle (red) visible only in the front camera. Each LOPR sample captures a realistic hypothetical evolution of the scene, such as variations in the velocity of the oncoming car. (Right) Correct forecasting of an oncoming vehicle and a static road layout (orange). Both examples demonstrate that our framework is capable of multi-future reasoning and leveraging multi-modal observations.