MonoOcc: Digging into Monocular Semantic Occupancy Prediction

TL;DR

This work tackles monocular semantic occupancy prediction (SSC) by addressing optimization and information bottlenecks in prior cascaded pipelines. It introduces MonoOcc, which combines an image-conditioned cross-attention mechanism and a 2D semantic auxiliary loss to improve early-stage feature learning, with a privileged branch that leverages a large pretrained image backbone and a Cross View Transformer to exploit temporal context. A distillation module then transfers rich temporal knowledge from the privileged branch to a lightweight monocular branch, yielding substantial gains on small and long-tailed objects while maintaining efficiency. On SemanticKITTI, MonoOcc achieves state-of-the-art SSC performance, demonstrating that augmenting monocular frameworks with cross-modal refinement and temporally enhanced privileged knowledge can match or exceed stronger baselines with favorable resource trade-offs. The approach offers practical significance for cost-efficient, camera-based 3D perception in autonomous driving.

Abstract

Monocular Semantic Occupancy Prediction aims to infer the complete 3D geometry and semantic information of scenes from only 2D images. It has garnered significant attention, particularly due to its potential to enhance the 3D perception of autonomous vehicles. However, existing methods rely on a complex cascaded framework with relatively limited information to restore 3D scenes, including a dependency on supervision solely on the whole network's output, single-frame input, and the utilization of a small backbone. These challenges, in turn, hinder the optimization of the framework and yield inferior prediction results, particularly concerning smaller and long-tailed objects. To address these issues, we propose MonoOcc. In particular, we (i) improve the monocular occupancy prediction framework by proposing an auxiliary semantic loss as supervision to the shallow layers of the framework and an image-conditioned cross-attention module to refine voxel features with visual clues, and (ii) employ a distillation module that transfers temporal information and richer knowledge from a larger image backbone to the monocular semantic occupancy prediction framework with low cost of hardware. With these advantages, our method yields state-of-the-art performance on the camera-based SemanticKITTI Scene Completion benchmark. Codes and models can be accessed at https://github.com/ucaszyp/MonoOcc

Figures (3)

• Figure 1: Quantitative results of semantic occupancy prediction on SemanticKITTIbehley2019semantickitti test set compared with the state-of-the-art VoxFormer-Sli2023voxformer and OccFormerzhang2023occformer. Note that our method outperforms the latter methods in the SSC mIoU, while also achieving a significant boost on both small objects (bicycle, motorcycle, traffic-sign) and long-tailed objects (truck, other-vehicle, other-ground). Compared with VoxFormer-S, the relative percentage increase of our method on average performance, small objects and long-tailed objects are denoted by green, yellow and orange, respectively.
• Figure 2: The architecture of our proposed framework (see section \ref{['sec:method']} for details)
• Figure 3: Qualitative results of our method and VoxFormer on SemanticKITTI dataset