TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancy

TL;DR

TACOcc tackles two core challenges in 3D semantic occupancy: geometry–semantics mismatch from fixed fusion and surface detail loss under sparse annotations. It introduces a target-scale adaptive bidirectional symmetric retrieval for cross-modal fusion and a multi-modal volume rendering pipeline based on 3D Gaussian Splatting to supervise surface details. The framework jointly optimizes cross-modal alignment and 2D–3D consistency with photometric supervision and Gaussian parameter consistency, achieving state-of-the-art results on nuScenes (28.4% mIoU) and outperforming prior methods. The method improves small-object reconstruction and robustness across challenging conditions, while maintaining reasonable efficiency due to adaptive neighborhood sizing.

Abstract

The performance of multi-modal 3D occupancy prediction is limited by ineffective fusion, mainly due to geometry-semantics mismatch from fixed fusion strategies and surface detail loss caused by sparse, noisy annotations. The mismatch stems from the heterogeneous scale and distribution of point cloud and image features, leading to biased matching under fixed neighborhood fusion. To address this, we propose a target-scale adaptive, bidirectional symmetric retrieval mechanism. It expands the neighborhood for large targets to enhance context awareness and shrinks it for small ones to improve efficiency and suppress noise, enabling accurate cross-modal feature alignment. This mechanism explicitly establishes spatial correspondences and improves fusion accuracy. For surface detail loss, sparse labels provide limited supervision, resulting in poor predictions for small objects. We introduce an improved volume rendering pipeline based on 3D Gaussian Splatting, which takes fused features as input to render images, applies photometric consistency supervision, and jointly optimizes 2D-3D consistency. This enhances surface detail reconstruction while suppressing noise propagation. In summary, we propose TACOcc, an adaptive multi-modal fusion framework for 3D semantic occupancy prediction, enhanced by volume rendering supervision. Experiments on the nuScenes and SemanticKITTI benchmarks validate its effectiveness.

Figures (7)

• Figure 1: Overview of proposed TACOcc. The point clouds and images undergo feature extraction and dimensional transformation to generate sparse voxel features. These features are then precisely aligned and fused through an adaptive fusion module (See Sec.\ref{['sec:3.1']}), followed by enhancement via a volume rendering optimization module (See Sec.\ref{['sec:3.2']}), thereby improving the performance of 3D semantic occupancy prediction.
• Figure 2: Adaptive Fuser. To achieve target-aligned fusion of sparse voxel features from two modalities, a dynamic $k$ selection module generates an optimal $k$ value for each query. Based on this value, bidirectional symmetric retrieval is performed. The retrieved keys are stacked and passed through feature extraction and non-linear transformation to compute attention weights. These weights are then used to weight the current query, resulting in the fused feature.
• Figure 3: Qualitative comparison results on the nuScenes validation set. The top left displays the input surrounding images, while the bottom left shows the occupancy predictions made by our TACOcc for each surrounding image. Subsequently visualized are the driver's perspective from our TACOcc method, the Bird-Eye View (BEV) of M-CONet wang2023openoccupancy, the BEV of Co-Occ pan2024co, the BEV of our TACOcc method, and the ground truth (GT) occupancy from SurroundOcc wei2023surroundocc.
• Figure 4: Scenarios with more small targets.
• Figure 5: Scenarios with more large targets.