GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding

TL;DR

This work tackles open-vocabulary 3D object affordance grounding by introducing GREAT, aGeometry-Intention Collaborative Inference framework that reasons about invariant object geometry and potential interaction intentions through multi-step chain-of-thought reasoning. It combines multi-modal knowledge with both point cloud and image data via a Multi-Head Affordance Chain-of-Thought (MHACoT) and a Cross-Modal Adaptive Fusion Module (CMAFM), enabling robust grounding beyond fixed vocabularies. The authors also present PIADv2, a large 3D affordance dataset with diverse object categories and interaction images to support open-vocabulary evaluation. Experiments show state-of-the-art performance across seen and unseen object and affordance partitions, demonstrating strong generalization and practical potential for robot perception and manipulation in unknown environments.

Abstract

Open-Vocabulary 3D object affordance grounding aims to anticipate ``action possibilities'' regions on 3D objects with arbitrary instructions, which is crucial for robots to generically perceive real scenarios and respond to operational changes. Existing methods focus on combining images or languages that depict interactions with 3D geometries to introduce external interaction priors. However, they are still vulnerable to a limited semantic space by failing to leverage implied invariant geometries and potential interaction intentions. Normally, humans address complex tasks through multi-step reasoning and respond to diverse situations by leveraging associative and analogical thinking. In light of this, we propose GREAT (GeometRy-intEntion collAboraTive inference) for Open-Vocabulary 3D Object Affordance Grounding, a novel framework that mines the object invariant geometry attributes and performs analogically reason in potential interaction scenarios to form affordance knowledge, fully combining the knowledge with both geometries and visual contents to ground 3D object affordance. Besides, we introduce the Point Image Affordance Dataset v2 (PIADv2), the largest 3D object affordance dataset at present to support the task. Extensive experiments demonstrate the effectiveness and superiority of GREAT. The code and dataset are available at https://yawen-shao.github.io/GREAT/.

Figures (8)

• Figure 1: Difference and Motivation.(a) Object affordance grounding on seen setting. (b) Open-Vocabulary Affordance Grounding (OVAG) with previous paradigms. (c) Observing interaction images, people engage in brainstorming through memory representations, drawing on prior interaction experiences to perform analogical reasoning and infer appropriate actions. (d) OVAG with our geometry-intention collaborative inference with chain-of-thought, step-by-step identifies the interaction part, extracts geometric attributes, reasons about corresponding interaction and brainstorms underlying interaction intentions, jointly grounding the 3D object affordance.
• Figure 2: GREAT pipeline. Initially, it extracts the respective features $\mathbf{F}_{i}, \mathbf{F}_{p}$ through modality-specific backbones, then results of MHACoT inference are encoded and aggregated to form object/affordance knowledge features $\bar{\mathbf{T}}_{o}, \bar{\mathbf{T}}_{a}$ (Sec. \ref{['Sec.3.2']}). Next, GREAT utilizes CMAFM to inject knowledge into $\mathbf{F}_{p}$ and $\mathbf{F}_{i}$ is directly fused to obtain fusion features $\mathbf{F}_{tp},\mathbf{F}_{ti}$ (Sec. \ref{['Sec.3.3']}). Eventually, $\mathbf{F}_{tp}$ and $\mathbf{F}_{ti}$ are sent to the decoder to ground 3D object affordance $\phi$ (Sec. \ref{['Sec.3.4']}).
• Figure 3: PIADv2 Dataset.(a) Extensive data examples from PIADv2, the red region in point clouds is the affordance annotation. (b) Category distribution in PIADv2. (c) Confusion matrix between affordance and object categories, where the horizontal axis represents object category and the vertical axis represents affordance category. (d) Ratio of images and point clouds in each object category.
• Figure 4: Visualization Results. The first row is the interaction image and the last row is the ground truth of 3D object affordance in point cloud. The left-middle-right partitions correspond to the visual comparison results for different 3D object affordance in the Seen, Unseen Object, and Unseen Affordance partitions, respectively. The depth of red represents the affordance probability.
• Figure 5: Visual Attention.(a) Attention heatmaps on geometries with ($\bm{w}$) and without ($\bm{w/o}$) the AffCoT. (b) Feature maps on interaction images with ($\bm{w}$) and without ($\bm{w/o}$) the ObjCoT.