Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications

TL;DR

Cam4DOcc introduces the first benchmark for camera-only 4D occupancy forecasting in autonomous driving, combining a new dataset format derived from nuScenes, nuScenes-Occupancy, and Lyft-Level5 with a standardized four-task evaluation protocol. It provides four baselines and a novel end-to-end network, OCFNet, that jointly forecasts present and near-future occupancy and 3D backward centripetal flow. Across extensive experiments, OCFNet consistently outperforms baselines in near-term forecast accuracy, demonstrating the value of end-to-end spatiotemporal modeling and flow guidance for 4D occupancy prediction. The work highlights the feasibility and potential of purely camera-based 4D occupancy forecasting and lays a foundation for future research in longer horizons and more efficient training regimes.

Abstract

Understanding how the surrounding environment changes is crucial for performing downstream tasks safely and reliably in autonomous driving applications. Recent occupancy estimation techniques using only camera images as input can provide dense occupancy representations of large-scale scenes based on the current observation. However, they are mostly limited to representing the current 3D space and do not consider the future state of surrounding objects along the time axis. To extend camera-only occupancy estimation into spatiotemporal prediction, we propose Cam4DOcc, a new benchmark for camera-only 4D occupancy forecasting, evaluating the surrounding scene changes in a near future. We build our benchmark based on multiple publicly available datasets, including nuScenes, nuScenes-Occupancy, and Lyft-Level5, which provides sequential occupancy states of general movable and static objects, as well as their 3D backward centripetal flow. To establish this benchmark for future research with comprehensive comparisons, we introduce four baseline types from diverse camera-based perception and prediction implementations, including a static-world occupancy model, voxelization of point cloud prediction, 2D-3D instance-based prediction, and our proposed novel end-to-end 4D occupancy forecasting network. Furthermore, the standardized evaluation protocol for preset multiple tasks is also provided to compare the performance of all the proposed baselines on present and future occupancy estimation with respect to objects of interest in autonomous driving scenarios. The dataset and our implementation of all four baselines in the proposed Cam4DOcc benchmark will be released here: https://github.com/haomo-ai/Cam4DOcc.

Figures (11)

• Figure 1: Cam4DOcc focuses on providing a novel dataset format, creating baselines modified from off-the-shelf camera-based perception and prediction approaches, and proposing a standardized evaluation protocol for the 4D occupancy forecasting task.
• Figure 2: Overall pipeline of constructing dataset in our Cam4DOcc based on the original nuScenes and nuScenes-Occupancy. The dataset is reorganized into a novel format that considers both general movable and static categories for the unified 4D occupancy forecasting task.
• Figure 3: Four types of baselines are proposed in the Cam4DOcc benchmark from the extension of occupancy prediction, point cloud prediction, and 2D instance prediction, as well as our end-to-end 4D occupancy forecasting network.
• Figure 4: System overview of our proposed OCFNet.
• Figure 5: Visualization of forecasting inflated GMO by our proposed OCFNet. The prediction results and ground truth from timestamps 1 to $N_f$ are assigned colors from dark to light. The motion trend of each moving object is represented by red arrows.