Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion

Rongtao Xu, Jinzhou Lin, Jialei Zhou, Jiahua Dong, Changwei Wang, Ruisheng Wang, Li Guo, Shibiao Xu, Xiaodan Liang

TL;DR

The paper addresses the challenge of accurate camera-based semantic scene completion (SSC) by exploiting rich multi-level image representations and knowledge from large vision models. It introduces CIGOcc, a two-stage framework with Deformable Multi-Level Fusion Network (DMFNet) for depth- and segmentation-based fusion and Complementary Information Guided Voxel Generation Network (CIGNet) for distilling Grounded-SAM knowledge into graphics feature fusion via deformable attention. The method achieves state-of-the-art mean IoU on SemanticKITTI for camera-based SSC, with pronounced improvements on small and long-tailed objects, while maintaining acceptable training efficiency. Overall, the work enhances 2D-to-3D reconstruction by effectively integrating depth, segmentation, and high-level semantics, offering practical benefits for autonomous driving perception systems.

Abstract

Camera-based occupancy prediction is a mainstream approach for 3D perception in autonomous driving, aiming to infer complete 3D scene geometry and semantics from 2D images. Almost existing methods focus on improving performance through structural modifications, such as lightweight backbones and complex cascaded frameworks, with good yet limited performance. Few studies explore from the perspective of representation fusion, leaving the rich diversity of features in 2D images underutilized. Motivated by this, we propose \textbf{CIGOcc, a two-stage occupancy prediction framework based on multi-level representation fusion. \textbf{CIGOcc extracts segmentation, graphics, and depth features from an input image and introduces a deformable multi-level fusion mechanism to fuse these three multi-level features. Additionally, CIGOcc incorporates knowledge distilled from SAM to further enhance prediction accuracy. Without increasing training costs, CIGOcc achieves state-of-the-art performance on the SemanticKITTI benchmark. The code is provided in the supplementary material and will be released https://github.com/VitaLemonTea1/CIGOcc

Complementary Information Guided Occupancy Prediction via Multi-Level Representation Fusion

TL;DR

The paper addresses the challenge of accurate camera-based semantic scene completion (SSC) by exploiting rich multi-level image representations and knowledge from large vision models. It introduces CIGOcc, a two-stage framework with Deformable Multi-Level Fusion Network (DMFNet) for depth- and segmentation-based fusion and Complementary Information Guided Voxel Generation Network (CIGNet) for distilling Grounded-SAM knowledge into graphics feature fusion via deformable attention. The method achieves state-of-the-art mean IoU on SemanticKITTI for camera-based SSC, with pronounced improvements on small and long-tailed objects, while maintaining acceptable training efficiency. Overall, the work enhances 2D-to-3D reconstruction by effectively integrating depth, segmentation, and high-level semantics, offering practical benefits for autonomous driving perception systems.

Abstract

Camera-based occupancy prediction is a mainstream approach for 3D perception in autonomous driving, aiming to infer complete 3D scene geometry and semantics from 2D images. Almost existing methods focus on improving performance through structural modifications, such as lightweight backbones and complex cascaded frameworks, with good yet limited performance. Few studies explore from the perspective of representation fusion, leaving the rich diversity of features in 2D images underutilized. Motivated by this, we propose \textbf{CIGOcc, a two-stage occupancy prediction framework based on multi-level representation fusion. \textbf{CIGOcc extracts segmentation, graphics, and depth features from an input image and introduces a deformable multi-level fusion mechanism to fuse these three multi-level features. Additionally, CIGOcc incorporates knowledge distilled from SAM to further enhance prediction accuracy. Without increasing training costs, CIGOcc achieves state-of-the-art performance on the SemanticKITTI benchmark. The code is provided in the supplementary material and will be released https://github.com/VitaLemonTea1/CIGOcc
Paper Structure (15 sections, 7 equations, 3 figures, 6 tables)

This paper contains 15 sections, 7 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Scene Completion
  4. Camera-based 3D Perception
  5. 3D Occupancy Prediction
  6. Method
  7. Deformable Multi-Level Fusion Network
  8. Complementary Information Guided Voxel Generation Network
  9. Training Loss
  10. Experiment
  11. Experimental Setup
  12. Comparison with Other Methods and Results
  13. Ablation Study
  14. Model efficiency
  15. Conclusion

Figures (3)

  • Figure 1: Quantitative results of semantic occupancy prediction performance, small objects performance and long-tailed performance. Compared to VoxFormer-T, our model has a significant improvement in mIoU.
  • Figure 2: Framework of CIGOcc. The input image is processed by Grounded-SAM to generate semantic features and segmentation masks, while the depth estimation network produces a depth map. DMFNet fuses the depth map and semantic features to generate initial voxel features and query proposals. For CIGNet, the image features extracted by ResNet, along with the query proposals, generate the voxel space via deformable cross-attention, which is then fused with DMFNet voxel features and enhanced through self-attention. Finally, the Occupancy Head performs occupancy prediction.
  • Figure 3: Qualitative results of our method and others. We performed a visual comparison with three other models, and it can be seen that our model achieves more precise segmentation of scene voxels with less voxel overlapping, while also being more accurate in road prediction.