OpenGround: Active Cognition-based Reasoning for Open-World 3D Visual Grounding

TL;DR

OpenGround addresses open-world 3D visual grounding by removing reliance on a fixed Object Lookup Table (OLT) through Active Cognition-based Reasoning (ACR). ACR builds a cognitive task chain to progressively ground targets and employs Active Cognition Enhancement (ACE) to dynamically extend cognition and the OLT via active perception and 2D-to-3D lifting. The authors introduce OpenTarget, a 7,724-description dataset built on ScanNet++ and Articulate3D to simulate unseen objects, and demonstrate competitive zero-shot performance on Nr3D/ScanRefer while achieving state-of-the-art open-world grounding on OpenTarget. Across extensive ablations, OpenGround shows robustness to VLM size, flexibility to integrate with existing methods, and clear benefits from human-like task planning and progressive context-aware grounding. These results suggest a practical, scalable path toward true open-world 3D grounding in real environments.

Abstract

3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes. Existing methods rely on a pre-defined Object Lookup Table (OLT) to query Visual Language Models (VLMs) for reasoning about object locations, which limits the applications in scenarios with undefined or unforeseen targets. To address this problem, we present OpenGround, a novel zero-shot framework for open-world 3D visual grounding. Central to OpenGround is the Active Cognition-based Reasoning (ACR) module, which is designed to overcome the fundamental limitation of pre-defined OLTs by progressively augmenting the cognitive scope of VLMs. The ACR module performs human-like perception of the target via a cognitive task chain and actively reasons about contextually relevant objects, thereby extending VLM cognition through a dynamically updated OLT. This allows OpenGround to function with both pre-defined and open-world categories. We also propose a new dataset named OpenTarget, which contains over 7000 object-description pairs to evaluate our method in open-world scenarios. Extensive experiments demonstrate that OpenGround achieves competitive performance on Nr3D, state-of-the-art on ScanRefer, and delivers a substantial 17.6% improvement on OpenTarget. Project Page at [this https URL](https://why-102.github.io/openground.io/).

Figures (15)

• Figure 1: Comparative overview of previous zero-shot methods' paradigm and ours. Previous paradigm is single-step grounding based on predefined $\mathcal{OLT}$, unable to ground undefined objects. In contrast, our method introduces Active Cognition-based Reasoning to the paradigm, which enhances cognition before each grounding step and grounds target object progressively.
• Figure 2: Data Collection Pipeline. The pipeline generates discriminative object descriptions via three annotation stages and two-stage verification. It leverages hierarchical labels (e.g., cabinet→drawer→handle), selects target–distractor views, and employs VLMs for context-aware descriptions using parent annotations. Quality is ensured via VLM voting and manual refinement.
• Figure 3: Overview of the OpenGround framework. The core of our framework is the Active Cognition-based Reasoning (ACR) module. First, the ACR invokes Cognitive Task Chain Construction module to obtain a sequential task chain to guide step-by-step grounding. Next, the ACR module progresses along the task chain to ground objects progressively. For objects not present in the $\mathcal{OLT}$, it activates the Active Cognition Enhancement module to extend the $\mathcal{OLT}$ with newly perceived objects around previously grounded objects. Then, the ACR module uses Single-Step Grounding which prompts VLM with annotated images from perspectives focused on candidates (with reference to previously grounded objects) to obtain the target object's ID in this step. The ID is used to retrieve the object's 3D bounding box from the extended $\mathcal{OLT}$. Upon completing the ACR module's workflow, we obtain the bounding box of the final target.
• Figure 4: (a) illustrates three sequential steps (Objects Parsing, Objects Retrieval, Task Chain Construction) and contrasts human-like reasoning. (b) shows the edit distances between constructed and human given task chains.
• Figure 5: Illustration of perspective selection strategies in ACE.