LOC: A General Language-Guided Framework for Open-Set 3D Occupancy Prediction

TL;DR

LOC tackles open-set 3D occupancy prediction by translating 2D vision-language knowledge into dense 3D voxel space via a general framework that combines a dense occupancy head with a CLIP-aligned language head. It introduces a Robust Densification Strategy, including Dynamic-Static Separation and Poisson Reconstruction, to generate dense voxel representations from sparse LiDAR and multi-frame data, followed by Dense Contrastive Learning (DCL) that aligns voxel features with CLIP text embeddings through predefined prompts. The method can optionally leverage existing 3D occupancy ground truth to reinforce learning via a knowledge-distillation loss $L_{KD}$ and a contrastive loss $L_{DCL}$, with a total objective $L = L_{CE} + \lambda_1 L_{KD} + \lambda_2 L_{DCL}$. Experiments on nuScenes demonstrate superior open-set recognition and competitive closed-set accuracy, highlighting the approach's ability to detect unknown objects without extra training data and its potential to extend open-vocabulary 3D perception in autonomous driving.

Abstract

Vision-Language Models (VLMs) have shown significant progress in open-set challenges. However, the limited availability of 3D datasets hinders their effective application in 3D scene understanding. We propose LOC, a general language-guided framework adaptable to various occupancy networks, supporting both supervised and self-supervised learning paradigms. For self-supervised tasks, we employ a strategy that fuses multi-frame LiDAR points for dynamic/static scenes, using Poisson reconstruction to fill voids, and assigning semantics to voxels via K-Nearest Neighbor (KNN) to obtain comprehensive voxel representations. To mitigate feature over-homogenization caused by direct high-dimensional feature distillation, we introduce Densely Contrastive Learning (DCL). DCL leverages dense voxel semantic information and predefined textual prompts. This efficiently enhances open-set recognition without dense pixel-level supervision, and our framework can also leverage existing ground truth to further improve performance. Our model predicts dense voxel features embedded in the CLIP feature space, integrating textual and image pixel information, and classifies based on text and semantic similarity. Experiments on the nuScenes dataset demonstrate the method's superior performance, achieving high-precision predictions for known classes and distinguishing unknown classes without additional training data.

Figures (4)

• Figure 1: Given a set of images containing a previously-unknown object (left), the closed-set occupancy model classifies the voxels belonging to that object as either a known category or as free (center, black circle). Our goal is to predict known classes and identify unknown classes.
• Figure 2: Architecture of the proposed model. Input images are first transformed into 3D voxel features by the occupancy network M. Features are then fed into two parallel decoding heads: Occupancy Head performs occupancy prediction, where the dense occupancy information obtained from the Robust Densification Strategy serves as the primary supervision signal for the Occupancy Head; Language Head aligns voxel features with CLIP text embeddings using distillation loss and DCL loss.
• Figure 3: Results from the nuScenes dataset, where the unknown class is the construction vehicle
• Figure 4: 3D feature space sparsity