Integrating Specialized and Generic Agent Motion Prediction with Dynamic Occupancy Grid Maps

TL;DR

This work tackles robust driving-scene motion forecasting by unifying agent-specific (vehicle-centric) and agent-agnostic (occupancy-based) predictions. It introduces a multi-head architecture that jointly predicts current vehicle/flow grids, future vehicle/flow grids, and future DOGMs, all guided by a flow-aware, interdependent loss and a conditional variational backbone. The system leverages DOGMs and camera-based vehicle segmentation to form a rich BEV representation, enabling accurate predictions even under occlusion and for unrecognized agents. Evaluations on nuScenes and the Woven Planet dataset show that flow-guided occupancy predictions improve both vehicle and generic dynamic predictions, while maintaining real-time performance with around 80 ms end-to-end latency on a single V100 GPU. Overall, the approach advances holistic scene understanding by enforcing cross-task consistency and enabling safe, diverse future predictions in complex urban environments.

Abstract

Accurate prediction of driving scene is a challenging task due to uncertainty in sensor data, the complex behaviors of agents, and the possibility of multiple feasible futures. Existing prediction methods using occupancy grid maps primarily focus on agent-agnostic scene predictions, while agent-specific predictions provide specialized behavior insights with the help of semantic information. However, both paradigms face distinct limitations: agent-agnostic models struggle to capture the behavioral complexities of dynamic actors, whereas agent-specific approaches fail to generalize to poorly perceived or unrecognized agents; combining both enables robust and safer motion forecasting. To address this, we propose a unified framework by leveraging Dynamic Occupancy Grid Maps within a streamlined temporal decoding pipeline to simultaneously predict future occupancy state grids, vehicle grids, and scene flow grids. Relying on a lightweight spatiotemporal backbone, our approach is centered on a tailored, interdependent loss function that captures inter-grid dependencies and enables diverse future predictions. By using occupancy state information to enforce flow-guided transitions, the loss function acts as a regularizer that directs occupancy evolution while accounting for obstacles and occlusions. Consequently, the model not only predicts the specific behaviors of vehicle agents, but also identifies other dynamic entities and anticipates their evolution within the complex scene. Evaluations on real-world nuScenes and Woven Planet datasets demonstrate superior prediction performances for dynamic vehicles and generic dynamic scene elements compared to baseline methods.

Figures (7)

• Figure 1: Our network generates detection and prediction grids, for agent-specific and agent-agnostic behaviors. Scene flow grids play a central role in bridging these two categories, leveraging flow-guided occupancy predictions and optimizing them through tailored loss functions.
• Figure 2: Proposed prediction framework: Input sequence of vehicle grids, DOGM occupancy state grids and associated velocity grids capture the past scene evolution. The network takes inspiration from video prediction, conditional variational approach, skip connection architecture and predicts i) vehicle and dynamic grid at current timestep in the detection head $\mathbf{Z}^{\text{det}}$, ii) sequence of future vehicle and scene flow grids in the prediction $\mathbf{Z}_{1:T}^{\text{pred}}$ and iii) sequence of three-channel occupancy state grids in the DOGM prediction head $\mathbf{Z}_{1:T}^{\text{ogm}}$.
• Figure 3: Scene Representation: DOGM state grids and velocity grids $\{\mathbf{O}, \mathbf{V}\}$ are generated from LiDAR data. To incorporate semantic information, BEV features encoded from camera images are fused with occupancy state grids to predict vehicle segmentation grids $\mathbf{S}_t$.
• Figure 4: Flow-guided occupancy prediction: The warping operation $f_w(\cdot)$ propagates cell occupancies (shown in purple) from the initial detection grid $\mathbf{D}_{\text{det}}^{\text{veh}}$ based on the motion vectors, that point to the origin of their respective occupancy in the backward flow grid $\mathbf{P}_{1}^{\text{flow}}$. The figure illustrates the first step ($\tau=1$) of the recursive process.
• Figure 5: Prediction example of a driving scene from the Nuscenes Datatset. The scene comprises two dynamic vehicles, multiple pedestrians and the ego-vehicle in the grid center. Input sequence comprises of the vehicle grids, DOGM state and velocity grids. Prediction results are displayed in the left five columns while right three show the corresponding ground truth grids. Detection head $\mathbf{Z}^{\text{det}}$ grids are displayed for the current timestep (t=0.0s) row while the remaining rows show prediction head $\mathbf{Z}^{\text{pred}}$ and DOGM prediction head $\mathbf{Z}^{\text{ogm}}$ grids.