AGO: Adaptive Grounding for Open World 3D Occupancy Prediction

TL;DR

AGO tackles open-world 3D occupancy prediction by integrating grounding with noise prompts and a modality adapter to bridge VLM image embeddings and 3D voxel features. The method combines closed-world grounding with 3D pseudo-labels derived from multi-frame VLM masks and an adaptive alignment that maps 3D embeddings to VLM image space, guided by an open-world identifier that uses information-theoretic criteria to handle known and unknown objects. Empirically, AGO achieves state-of-the-art closed-world self-supervised performance on Occ3D-nuScenes and demonstrates strong zero-shot and few-shot open-world transfer, significantly outperforming prior alignment-based and grounding-only approaches. The work delivers a practical, parameter-efficient framework for robust 3D scene understanding in autonomous driving, with broad implications for open-vocabulary 3D perception in dynamic environments.

Abstract

Open-world 3D semantic occupancy prediction aims to generate a voxelized 3D representation from sensor inputs while recognizing both known and unknown objects. Transferring open-vocabulary knowledge from vision-language models (VLMs) offers a promising direction but remains challenging. However, methods based on VLM-derived 2D pseudo-labels with traditional supervision are limited by a predefined label space and lack general prediction capabilities. Direct alignment with pretrained image embeddings, on the other hand, often fails to achieve reliable performance because of inconsistent image and text representations in VLMs. To address these challenges, we propose AGO, a novel 3D occupancy prediction framework with adaptive grounding to handle diverse open-world scenarios. AGO first encodes surrounding images and class prompts into 3D and text embeddings, respectively, leveraging similarity-based grounding training with 3D pseudo-labels. Additionally, a modality adapter maps 3D embeddings into a space aligned with VLM-derived image embeddings, reducing modality gaps. Experiments on Occ3D-nuScenes show that AGO improves unknown object prediction in zero-shot and few-shot transfer while achieving state-of-the-art closed-world self-supervised performance, surpassing prior methods by 4.09 mIoU. Code is available at: https://github.com/EdwardLeeLPZ/AGO.

Figures (10)

• Figure 1: Open-world 3D semantic occupancy prediction. (a) Supervision based on pseudo labels with fixed classes cannot predict novel categories, such as the "construction vehicle" and "vegetation". (b) Similarity-based alignment suffers from significant mismatches due to issues like modality discrepancy, leading to confusions between e.g."sidewalk" and "driveable surface". (c) Our proposed Adaptive Grounding flexibly accommodates both known and unknown objects, achieving more precise open world occupancy prediction.
• Figure 2: Modality gaps between image and text embeddings in pre-trained VLMs. The similarity heatmap generated using the image and prompt "car" only covers a small range of about 0.1 (with a peak of only 0.36), while the low resolution of the embeddings also results in spatial misalignment and semantic ambiguity.
• Figure 3: AGO architecture. The upper part illustrates the generation process of 3D pseudo-labels and image embeddings based on pre-trained VLMs during training. The lower part depicts the main architecture of our AGO framework, which comprises a frozen pre-trained text encoder, a vision-centric 3D encoder, a modality adapter, and an open-world identifier. The detailed illustration in the middle showcases our training paradigm, which consists of a noise-augmented grounding training and an adaptive image embedding alignment.
• Figure 4: Visualization of self-supervised 3D semantic occupancy prediction on the Occ3D-nuScenes occupancy benchmark. Our method demonstrates more detailed predictions for dynamic ("pedestrian") and long-tailed ("barrier") objects.
• Figure 5: Visualization of Open-world Zero-&Few-Shot Transfer. Our method can not only adapt to category changes from coarse to fine but also easily accommodate newly introduced, previously unknown categories with only a small amount of few-shot fine-tuning.