Spatiotemporal Decoupling for Efficient Vision-Based Occupancy Forecasting

TL;DR

A novel spatiotemporal decoupling vision-based paradigm to explicitly tackle the bias and achieve both effective and efficient 3D OCF is proposed and an efficient multi-head network EfficientOCF is developed to achieve 3D OCF with the devised spatiotemporally decoupled representation.

Abstract

The task of occupancy forecasting (OCF) involves utilizing past and present perception data to predict future occupancy states of autonomous vehicle surrounding environments, which is critical for downstream tasks such as obstacle avoidance and path planning. Existing 3D OCF approaches struggle to predict plausible spatial details for movable objects and suffer from slow inference speeds due to neglecting the bias and uneven distribution of changing occupancy states in both space and time. In this paper, we propose a novel spatiotemporal decoupling vision-based paradigm to explicitly tackle the bias and achieve both effective and efficient 3D OCF. To tackle spatial bias in empty areas, we introduce a novel spatial representation that decouples the conventional dense 3D format into 2D bird's-eye view (BEV) occupancy with corresponding height values, enabling 3D OCF derived only from 2D predictions thus enhancing efficiency. To reduce temporal bias on static voxels, we design temporal decoupling to improve end-to-end OCF by temporally associating instances via predicted flows. We develop an efficient multi-head network EfficientOCF to achieve 3D OCF with our devised spatiotemporally decoupled representation. A new metric, conditional IoU (C-IoU), is also introduced to provide a robust 3D OCF performance assessment, especially in datasets with missing or incomplete annotations. The experimental results demonstrate that EfficientOCF surpasses existing baseline methods on accuracy and efficiency, achieving state-of-the-art performance with a fast inference time of 82.33ms with a single GPU. Our code will be released as open source.

Figures (9)

• Figure 1: Visualization of TP (green), FP (red), and FN (yellow) results against ground truth at continuous timesteps before and after instance-aware refinement.
• Figure 2: Visualization of OCF results of EfficientOCF, OCFNet, and ground truth from the nuScenes-Occupancy dataset wang2023openoccupancy. The occupancy forecasting results and ground-truth labels from timesteps 0 to $N_\text{f}$ are assigned colors from dark to light.
• Figure II: Unlike the traditional 3D vision-based OCF pipeline, our proposed paradigm implements spatiotemporal decoupling, thus achieving the efficiency of 2D OCF methods with the instance-aware capability of 3D OCF approaches.
• Figure III: Architecture of our proposed EfficientOCF. The perception module (gray area) extracts 2D visual features from the input surrounding images and lifts them to 3D voxel features. Subsequently, the voxel features are further compressed into BEV features and aggregated by the aggregation module (yellow area). Next, the prediction module (blue area) with three branches sharing the same 2D structure, achieves initial OCF in 2D BEV and predicts backward centripetal flow and height values for BEV occupancy. The refinement module (green area) forecasts instances using estimated flow to refine initial 2D OCF results, which are ultimately lifted to 3D space.
• Figure IV: The process of refining initial OCF results following temporal decoupling. We exploit step-wise association along the time axis to improve forecasting quality.