Camera-Only 3D Panoptic Scene Completion for Autonomous Driving through Differentiable Object Shapes

TL;DR

This work tackles camera-only 3D panoptic scene completion for autonomous driving by introducing OffsetOcc, a framework that learns differentiable object shapes as sets of 3D offsets around object centers. An Object Module predicts per-object class, center, and a dense set of offsets, while a Panoptic Module merges these with a baseline occupancy decoder to produce panoptic voxel occupancy, trained in two stages with carefully designed Hungarian-based losses. On Occ3D-nuScenes, OffsetOcc achieves competitive 3D semantic occupancy metrics (e.g., 28.0 miou, 43.9 iou with mask; 17.2 miou, 24.9 iou without mask) and a panoptic lidar pq of 29.4, demonstrating the viability of object-centric, differentiable occupancy for occluded regions. The approach is lightweight, extensible to temporal reasoning, and accompanied by public code to facilitate further research in semantic and panoptic 3D scene understanding from monocular cues.

Abstract

Autonomous vehicles need a complete map of their surroundings to plan and act. This has sparked research into the tasks of 3D occupancy prediction, 3D scene completion, and 3D panoptic scene completion, which predict a dense map of the ego vehicle's surroundings as a voxel grid. Scene completion extends occupancy prediction by predicting occluded regions of the voxel grid, and panoptic scene completion further extends this task by also distinguishing object instances within the same class; both aspects are crucial for path planning and decision-making. However, 3D panoptic scene completion is currently underexplored. This work introduces a novel framework for 3D panoptic scene completion that extends existing 3D semantic scene completion models. We propose an Object Module and Panoptic Module that can easily be integrated with 3D occupancy and scene completion methods presented in the literature. Our approach leverages the available annotations in occupancy benchmarks, allowing individual object shapes to be learned as a differentiable problem. The code is available at https://github.com/nicolamarinello/OffsetOcc .

Figures (5)

• Figure 1: OffsetOcc. Our method OffsetOcc models object occupancy as a set prediction task, where objects are represented by their 3d center and a set of offsets that describe their shape. The set of offsets is learned from the dataset annotations. Classification head omitted for conciseness.
• Figure 2: Overview of our framework OffsetOcc. Our approach can be integrated with modern occupancy models and consists of two key components: an Object Module and a Panoptic Module. The Object Module utilizes features extracted by an uplifting encoder to detect objects in the voxel grid and predict their occupancy using the offset occupancy mechanism. The Panoptic Module then merges these object predictions with the occupancy grid output to produce a panoptic occupancy map. Classification head in the Object Module omitted for conciseness.
• Figure 3: Qualitative results on Occ3D-nuScenes validation set. Our model accurately differentiates and segments different objects in the scene.
• Figure 4: Effect of the voting radius. Increasing $r$ initially improves performance by smoothing predictions. However, beyond the optimal value of $9$, performance declines as the voting process starts to incorporate irrelevant voxels, leading to incorrect id assignments. Tested on the mini set of nuScenes.
• Figure 5: Additional qualitative results of our model OffsetOcc.