OccCylindrical: Multi-Modal Fusion with Cylindrical Representation for 3D Semantic Occupancy Prediction

TL;DR

OccCylindrical addresses the challenge of robust 3D semantic occupancy prediction by fusing multi-sensor data in cylindrical coordinates to align with LiDAR point distributions. The approach combines a depth-aware pseudo-3D camera point cloud, a depth-estimation module, and TPV-Polar-Fusion to generate three TPV planes refined by a shared encoder-decoder, yielding dense occupancy with improved fine-grained geometry. On nuScenes with SurroundOcc, OccCylindrical achieves state-of-the-art results, including in rainy and nighttime scenarios, and demonstrates strong performance on small dynamic objects while maintaining efficiency. The work advances practical perception for autonomous vehicles by preserving geometric details through cylindrical representations and targeted fusion strategies, enabling safer and more reliable scene understanding.

Abstract

The safe operation of autonomous vehicles (AVs) is highly dependent on their understanding of the surroundings. For this, the task of 3D semantic occupancy prediction divides the space around the sensors into voxels, and labels each voxel with both occupancy and semantic information. Recent perception models have used multisensor fusion to perform this task. However, existing multisensor fusion-based approaches focus mainly on using sensor information in the Cartesian coordinate system. This ignores the distribution of the sensor readings, leading to a loss of fine-grained details and performance degradation. In this paper, we propose OccCylindrical that merges and refines the different modality features under cylindrical coordinates. Our method preserves more fine-grained geometry detail that leads to better performance. Extensive experiments conducted on the nuScenes dataset, including challenging rainy and nighttime scenarios, confirm our approach's effectiveness and state-of-the-art performance. The code will be available at: https://github.com/DanielMing123/OccCylindrical

Figures (3)

• Figure 1: Pipeline for two approaches: Cartesian approach (top) and our approach (bottom). In contrast to the current multi-sensor fusion frameworks that process 3D feature volumes under Cartesian coordinates, we perform 3D semantic occupancy prediction through feature fusion across two distinct modalities under cylindrical coordinates. This takes into account the distribution of the sensor readings, preserving more fine-grained geometry information.
• Figure 2: Overall architecture of OccCylindrical. The surround-view images are initially processed through the 2D backbone to extract visual features. Subsequently, a DepthNet is employed to generate a depth distribution feature based on these visual features. The depth distribution feature is supervised using depth information from the LiDAR point cloud. Meanwhile, the predefined depth-distribution coordinate, used as a positional embedding, along with the depth feature, is fused back into the visual feature, resulting in a Depth-Aware context feature. The outer product operation is applied on the depth distribution feature and depth-aware context feature, resulting in a pseudo-3D point cloud. The TPV-Polar-Fusion module takes the pseudo-3D point cloud and the LiDAR point cloud as input to do the feature-level fusion and outputs three TPV-Polar planes. The shared encoder-decoder structure further refines the TPV-Polar planes and outputs TPV-Polar planes to the prediction head for the 3D semantic occupancy prediction.
• Figure 3: Qualitative study results for daytime, rainy, and nighttime scenarios displayed in the upper, middle, and bottom.