GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention

TL;DR

GaussianFormer3D presents a multi-modal, Gaussian-based framework for 3D semantic occupancy that leverages LiDAR priors and a LiDAR-guided 3D deformable attention mechanism to refine object-centric Gaussians in a lifted 3D space. The approach replaces dense voxel grids with a compact Gaussian representation and a Gaussian-to-voxel splatting module, enabling memory-efficient, high-resolution occupancy predictions. Extensive experiments across on-road and off-road datasets show competitive performance with state-of-the-art fusion methods while reducing memory usage, with clear gains on small, dynamic, and large surface classes. The combination of voxel-to-Gaussian initialization and 3D deformable attention underpins robust multi-modal fusion and efficient inference for autonomous driving scenarios.

Abstract

3D semantic occupancy prediction is critical for achieving safe and reliable autonomous driving. Compared to camera-only perception systems, multi-modal pipelines, especially LiDAR-camera fusion methods, can produce more accurate and detailed predictions. Although most existing works utilize a dense grid-based representation, in which the entire 3D space is uniformly divided into discrete voxels, the emergence of 3D Gaussians provides a compact and continuous object-centric representation. In this work, we propose a multi-modal Gaussian-based semantic occupancy prediction framework utilizing 3D deformable attention, named as GaussianFormer3D. We introduce a voxel-to-Gaussian initialization strategy to provide 3D Gaussians with geometry priors from LiDAR data, and design a LiDAR-guided 3D deformable attention mechanism for refining 3D Gaussians with LiDAR-camera fusion features in a lifted 3D space. We conducted extensive experiments on both on-road and off-road datasets, demonstrating that our GaussianFormer3D achieves high prediction accuracy that is comparable to state-of-the-art multi-modal fusion-based methods with reduced memory consumption and improved efficiency.

Figures (11)

• Figure 1: We propose a LiDAR-camera fusion-based semantic occupancy prediction framework named GaussianFormer3D. We use 3D Gaussians instead of dense grids to reduce memory consumption and enhance algorithm efficiency. GaussianFormer3D achieves comparable performance to state-of-the-art multi-modal occupancy methods with reduced memory usage.
• Figure 2: An overview of the proposed GaussianFormer3D framework. We first voxelize LiDAR point clouds to obtain non-empty voxel features for initializing the position and opacity of 3D Gaussians 3dgs. Then multi-scale LiDAR depth maps and camera feature maps are extracted through projection and an image backbone respectively, and multiplied via outer product to construct a lifted 3D fusion feature space. Gaussians are iteratively updated with 3D sparse convolution, 3D deformable attention, and property refinement. The Gaussian representation is eventually processed by a Gaussian-to-voxel splatting module gaussianformer to generate dense 3D semantic occupancy.
• Figure 3: Qualitative results on the on-road SurroundOcc surroundocc validation set. Our multi-modal Gaussian-based occupancy method can capture both semantics information and geometry structure of the surroundings. Best viewed on screen and in color.
• Figure 4: Qualitative results on the off-road WildOcc wildocc test set. Our multi-modal Gaussian-based occupancy method can outperform the ground truth (as shown in the first row) and predict classes such as puddle that are vital for off-road autonomous driving (as shown in the second row). Best viewed on screen and in color.
• Figure 5: Visualization comparison with GaussianFormer gaussianformer on SurroundOcc surroundocc. By incorporating LiDAR, our method can obtain Gaussians with more adaptive scales and shapes, resulting in more accurate semantic predictions and delicate geometry details.