COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction

TL;DR

The paper tackles vision-based 3D occupancy prediction by addressing geometry loss and heavy computation in dense occupancy representations. It introduces COTR, which combines a geometry-aware occupancy encoder with explicit-implicit view transformation to produce a compact $3$D OCC representation and a semantic-aware group decoder to boost semantic discriminability through coarse-to-fine grouping. The method achieves state-of-the-art results on Occ3D-nuScenes, with consistent IoU and mIoU gains across baselines and a substantial reduction in compute thanks to a compact representation of size $\tfrac{1}{16}$ of the original. By balancing geometric fidelity and semantic richness, COTR enhances robustness for autonomous driving perception while maintaining efficiency.

Abstract

The autonomous driving community has shown significant interest in 3D occupancy prediction, driven by its exceptional geometric perception and general object recognition capabilities. To achieve this, current works try to construct a Tri-Perspective View (TPV) or Occupancy (OCC) representation extending from the Bird-Eye-View perception. However, compressed views like TPV representation lose 3D geometry information while raw and sparse OCC representation requires heavy but redundant computational costs. To address the above limitations, we propose Compact Occupancy TRansformer (COTR), with a geometry-aware occupancy encoder and a semantic-aware group decoder to reconstruct a compact 3D OCC representation. The occupancy encoder first generates a compact geometrical OCC feature through efficient explicit-implicit view transformation. Then, the occupancy decoder further enhances the semantic discriminability of the compact OCC representation by a coarse-to-fine semantic grouping strategy. Empirical experiments show that there are evident performance gains across multiple baselines, e.g., COTR outperforms baselines with a relative improvement of 8%-15%, demonstrating the superiority of our method.

Figures (8)

• Figure 1: Left: Different representation for 3D perception. Right: The 3D Occupancy prediction results of different baselines with COTR on nuScenes tian2023occ3d. COTR outperforms baselines with a relative improvement of 8%-15%, demonstrating the superiority of our method.
• Figure 2: The overall architecture of COTR. $T$-frame surround-view images are first fed into the image featurizers to get the image features and depth distributions. Taking the image features and depth estimation as input, the geometry-aware occupancy encoder constructs a compact occupancy representation through efficient explicit-implicit view transformation. The semantic-aware group decoder utilizes a coarse-to-fine semantic grouping strategy cooperating with the Transformer-based mask classification to strongly strengthen the semantic discriminability of the compact occupancy representation.
• Figure 3: Proxy experiments. (a) depicts the comparison of different occupancy representations. The compact 3D OCC representation achieves a balance between performance and computational cost. (b) reports the per-class mIoU and distribution with and without using ground-truth semantic labels.
• Figure 4: Qualitative results comparison between baseline and our Geometry-aware Occupancy Encoder. The results demonstrate that the compact occupancy representation is able to capture more precise geometrical details for slimmer objects (e.g., pedestrians and poles) and is robust to occlusions by combining the advantages of implicit and explicit view transformation.
• Figure 5: The occupancy prediction results with label distribution. It is clear to see that the Semantic-aware Group Decoder (SGD) gives a big performance boost to the rare class.