STCOcc: Sparse Spatial-Temporal Cascade Renovation for 3D Occupancy and Scene Flow Prediction

TL;DR

STCOcc addresses the sparsity and spatial detail limitations of prior 3D occupancy and scene flow methods by introducing an explicit state-based framework that renovates 3D features using occupied-state guidance. The Spatial-Temporal Cascade Decoder combines a Self-Recursive Occupancy Predictor with Occlusion-Aware Temporal Self-Attention and Occupancy-Aware Spatial Cross-Attention, along with a sparse long-term temporal fusion, to achieve accurate geometry and dynamic modeling. The approach delivers state-of-the-art RayIoU and competitive mIoU/mAVE on Occ3D-nus and OpenOcc, while significantly reducing training memory to around 8–9 GB. This explicit, sparse, occupancy-guided renovation has strong practical potential for real-time 3D scene understanding in autonomous systems, enabling more reliable mapping and motion estimation with lower computational cost.

Abstract

3D occupancy and scene flow offer a detailed and dynamic representation of 3D scene. Recognizing the sparsity and complexity of 3D space, previous vision-centric methods have employed implicit learning-based approaches to model spatial and temporal information. However, these approaches struggle to capture local details and diminish the model's spatial discriminative ability. To address these challenges, we propose a novel explicit state-based modeling method designed to leverage the occupied state to renovate the 3D features. Specifically, we propose a sparse occlusion-aware attention mechanism, integrated with a cascade refinement strategy, which accurately renovates 3D features with the guidance of occupied state information. Additionally, we introduce a novel method for modeling long-term dynamic interactions, which reduces computational costs and preserves spatial information. Compared to the previous state-of-the-art methods, our efficient explicit renovation strategy not only delivers superior performance in terms of RayIoU and mAVE for occupancy and scene flow prediction but also markedly reduces GPU memory usage during training, bringing it down to 8.7GB. Our code is available on https://github.com/lzzzzzm/STCOcc

Figures (6)

• Figure 1: (a) Explicit versus Implicit Modeling: We propose a novel explicit state-based modeling approach that explicitly leverages the occupied state to maintain feature sparsity and model spatial details. (b) Comparison with Different Methods: Our approach achieves state-of-the-art performance of RayIoU and mAVE with lower training costs.
• Figure 2: The overall architecture of STCOcc. The STCOcc framework is primarily composed of four integral modules: a feature extractor that captures image features and depth distribution, a 3D coarse encoder that generates multi-resolution coarse voxel features, a multi-stage spatial-temporal cascade decoder that incrementally renovates these coarse voxel features in both spatial and temporal dimensions, and a head module designed to leverage the refined voxel features for the prediction of 3D occupancy and scene flow.
• Figure 3: Illustration of OA-SCA. Due to the projection process, sampled points along the same ray in the feature plane are assigned identical features, even when they represent empty voxel space, as depicted by the green points. To address this limitation, our approach integrates depth and occupancy information to assign appropriate weights to the sampled points, thereby enhancing the differentiation of features along each ray.
• Figure 4: Illustration of Sparse Temporal Fusion. We implement temporal fusion using a parallel strategy in a sparse manner, focusing only on modeling the sampled features.
• Figure 5: Ablation on the OA-SCA module. We visualize the features after refinement with and without the OA-SCA module.