LOPR: Latent Occupancy PRediction using Generative Models

TL;DR

LOPR tackles autonomous-vehicle environment prediction by learning a compact latent representation of multi-modal sensor data and performing stochastic future forecasting within that latent space. It decouples representation learning (via a $z \in \mathbb{R}^{64 \times 4 \times 4}$ VAE-GAN) from a transformer-based autoregressive predictor that introduces a stochastic latent $z_{stoch}$ to model multi-modal futures with a variational objective. The approach achieves state-of-the-art results on NuScenes and Waymo Open datasets, improves realism of predicted L-OGMs, and transfers readily between platforms, all while operating in real-time and without manual labeling. The framework supports conditioning on RGB and HD-map inputs and demonstrates robust predictions under partial observability, marking a practical advancement for planning and safety in autonomous systems.

Abstract

Environment prediction frameworks are integral for autonomous vehicles, enabling safe navigation in dynamic environments. LiDAR generated occupancy grid maps (L-OGMs) offer a robust bird's eye-view scene representation that facilitates joint scene predictions without relying on manual labeling unlike commonly used trajectory prediction frameworks. Prior approaches have optimized deterministic L-OGM prediction architectures directly in grid cell space. While these methods have achieved some degree of success in prediction, they occasionally grapple with unrealistic and incorrect predictions. We claim that the quality and realism of the forecasted occupancy grids can be enhanced with the use of generative models. We propose a framework that decouples occupancy prediction into: representation learning and stochastic prediction within the learned latent space. Our approach allows for conditioning the model on other available sensor modalities such as RGB-cameras and high definition maps. We demonstrate that our approach achieves state-of-the-art performance and is readily transferable between different robotic platforms on the real-world NuScenes, Waymo Open, and a custom dataset we collected on an experimental vehicle platform.

Figures (7)

• Figure 2: The illustration shows the LOPR framework which consists of (1) representation learning and (2) stochastic sequence prediction. In the representation learning stage, we train an encoder and decoder in an unsupervised manner. In the sequence prediction stage, we convert our OGM dataset to the low-dimensional representation, and perform training entirely in the latent space of our pre-trained generative model.
• Figure 3: Example reconstructions of OGMs, images, and maps from NuScenes.
• Figure 4: Predictions from NuScenes conditioned on L-OGM only. Ego vehicle is moving to the right. Moving objects are realistically propagated in the scene and the details of the static environment are maintained.
• Figure 5: Comparison of best predictions and random predictions from NuScenes conditioned on L-OGM only. Our framework is capable of inferring a previously unobserved oncoming agent entering the L-OGM in both scenes and different variations of static environment.
• Figure 6: Comparison between VAE-GAN and VAE on Nuscenes.