Global-Local Collaborative Inference with LLM for Lidar-Based Open-Vocabulary Detection

TL;DR

The paper tackles 3D open-vocabulary detection for lidar point clouds, addressing the reliance on object-level cues by introducing scene-level global context. It presents GLIS, a Global-Local Collaborative Inference framework that combines a 3D backbone for local features with a global branch and uses an Large Language Model to perform chain-of-thought reasoning to refine detections ($f_{loc}$/$f_{glob}$). The local branch employs BAOL and RPLG to generate high-quality proposals and pseudo labels, while the global branch educates scene understanding via scene captions supervised by MiniGPT-v2, enabling language-guided reasoning for misclassifications. Experiments on ScanNetV2 and SUN RGB-D show state-of-the-art improvements in mAP at IoU 0.25, demonstrating the value of integrating global scene cues and LLM-based reasoning in 3D OVD. Limitations stem from residual point-cloud noise and imperfect pseudo-labels, suggesting future work to further enhance robustness and cross-modal supervision.

Abstract

Open-Vocabulary Detection (OVD) is the task of detecting all interesting objects in a given scene without predefined object classes. Extensive work has been done to deal with the OVD for 2D RGB images, but the exploration of 3D OVD is still limited. Intuitively, lidar point clouds provide 3D information, both object level and scene level, to generate trustful detection results. However, previous lidar-based OVD methods only focus on the usage of object-level features, ignoring the essence of scene-level information. In this paper, we propose a Global-Local Collaborative Scheme (GLIS) for the lidar-based OVD task, which contains a local branch to generate object-level detection result and a global branch to obtain scene-level global feature. With the global-local information, a Large Language Model (LLM) is applied for chain-of-thought inference, and the detection result can be refined accordingly. We further propose Reflected Pseudo Labels Generation (RPLG) to generate high-quality pseudo labels for supervision and Background-Aware Object Localization (BAOL) to select precise object proposals. Extensive experiments on ScanNetV2 and SUN RGB-D demonstrate the superiority of our methods. Code is released at https://github.com/GradiusTwinbee/GLIS.

Figures (5)

• Figure 1: (a) The previous 3D OVD paradigm determines the class of an object proposal by comparing its point cloud feature with the class text features. As it only considers object-level/local information, the proposal may be wrongly recognized, e.g. mistaking the cabinet for a desk. (b) In contrast, we propose a Global-Local Collaborative Inference Scheme (GLIS) for 3D OVD, considering both the scene-level/global information and the object-level/local information. Additionally, we leverage the LLM to provide common sense for chain-of-thought inference, which can clarify the ambiguous object class step by step.
• Figure 2: The training pipeline of GLIS.
• Figure 3: The inference pipeline of GLIS.
• Figure 4: Visualizations of GLIS. The score of each proposal is the confidence that the proposal is truly a foreground object. These proposals, as well as their confidence scores, are generated by BAOL.
• Figure 5: Visualizations of detection results.