CurriFlow: Curriculum-Guided Depth Fusion with Optical Flow-Based Temporal Alignment for 3D Semantic Scene Completion

TL;DR

CurriFlow tackles camera-based 3D semantic scene completion by introducing three key strategies: optical-flow-guided temporal alignment (OFA2Net), curriculum-guided depth fusion (CDFNet), and semantic priors via Grounded-SAM distillation. The framework aligns multi-frame features with flow, gradually shifts depth supervision from sparse LiDAR to dense stereo during training, and injects high-level semantic priors to sharpen voxel-level learning, all while enforcing temporal-geometric consistency through deformable voxel generation and TPV-based cross-view losses. Experimental results on SemanticKITTI and KITTI360 demonstrate state-of-the-art performance and robust handling of occlusions and dynamic objects, validating the effectiveness of motion-guided fusion and curriculum learning for camera-only SSC. These contributions enable more reliable 3D scene understanding for autonomous driving in real-world, dynamic environments, with practical impact in perception pipelines and safety-critical planning.

Abstract

Semantic Scene Completion (SSC) aims to infer complete 3D geometry and semantics from monocular images, serving as a crucial capability for camera-based perception in autonomous driving. However, existing SSC methods relying on temporal stacking or depth projection often lack explicit motion reasoning and struggle with occlusions and noisy depth supervision. We propose CurriFlow, a novel semantic occupancy prediction framework that integrates optical flow-based temporal alignment with curriculum-guided depth fusion. CurriFlow employs a multi-level fusion strategy to align segmentation, visual, and depth features across frames using pre-trained optical flow, thereby improving temporal consistency and dynamic object understanding. To enhance geometric robustness, a curriculum learning mechanism progressively transitions from sparse yet accurate LiDAR depth to dense but noisy stereo depth during training, ensuring stable optimization and seamless adaptation to real-world deployment. Furthermore, semantic priors from the Segment Anything Model (SAM) provide category-agnostic supervision, strengthening voxel-level semantic learning and spatial consistency. Experiments on the SemanticKITTI benchmark demonstrate that CurriFlow achieves state-of-the-art performance with a mean IoU of 16.9, validating the effectiveness of our motion-guided and curriculum-aware design for camera-based 3D semantic scene completion.

Figures (5)

• Figure 1: Comparison between the previous and our SSC methods. (Top) The previous approach simply stacks multi-frame inputs and projects 2D features into voxel space through depth-based 2D-to-3D transformation, which suffers from depth noise and temporal misalignment. (Bottom) Our method explicitly maps and aligns temporal features via optical flow, guided by semantic supervision and curriculum-based depth fusion, enabling temporally consistent and semantically complete voxel generation.
• Figure 2: The overall CurriFlow architecture inputs three frames for depth, optical flow, and Grounded-SAM segmentation, extracting depth, flow, and instance masks. Instance masks assist the semantic loss to improve segmentation. The extracted image features, depth, and flow are temporally aligned by OFA2Net, fused via CDFNet, and further encoded by a spatial encoder for voxel-context modeling and multi-scale semantic aggregation.
• Figure 3: The OFA2Net aligns historical frame features via attention-based grid sampling, producing initial features $F_{\text{raw}}$. Occlusion mask gates error suppression, and neighborhood cross-attention fuses features into $F_{\text{fuse}}$.
• Figure 4: The CDFNet fuses stereo and monocular volumes through bi-directional attention, to produce depth volumes and $F_e$. Notice that DC and CDF is only active during training. DC: depth completion, CDF: curriculum-guided fusion
• Figure 5: Qualitative comparison of scene segmentation results. The first column shows the camera view, and the subsequent columns display the voxels outputs of Ground Truth, CurriFlow (Ours), CGFormer, and VoxFormer, respectively. Red boxes highlight key differences in segmentation accuracy across the methods.