Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

Dongqiang Gou, Xuming He

Abstract

Grounding natural language questions to functionally relevant regions in 3D objects -- termed language-driven 3D affordance grounding -- is essential for embodied intelligence and human-AI interaction. Existing methods, while progressing from label-based to language-driven approaches, still face challenges in open-vocabulary generalization, fine-grained geometric alignment, and part-level semantic consistency. To address these issues, we propose a novel two-stage cross-modal framework that enhances both semantic and geometric representations for open-vocabulary 3D affordance grounding. In the first stage, large language models generate part-aware instructions to recover missing semantics, enabling the model to link semantically similar affordances. In the second stage, we introduce two key components: Affordance Prototype Aggregation (APA), which captures cross-object geometric consistency for each affordance, and Intra-Object Relational Modeling (IORM), which refines geometric differentiation within objects to support precise semantic alignment. We validate the effectiveness of our method through extensive experiments on a newly introduced benchmark, as well as two existing benchmarks, demonstrating superior performance in comparison with existing methods.

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

Abstract

Grounding natural language questions to functionally relevant regions in 3D objects -- termed language-driven 3D affordance grounding -- is essential for embodied intelligence and human-AI interaction. Existing methods, while progressing from label-based to language-driven approaches, still face challenges in open-vocabulary generalization, fine-grained geometric alignment, and part-level semantic consistency. To address these issues, we propose a novel two-stage cross-modal framework that enhances both semantic and geometric representations for open-vocabulary 3D affordance grounding. In the first stage, large language models generate part-aware instructions to recover missing semantics, enabling the model to link semantically similar affordances. In the second stage, we introduce two key components: Affordance Prototype Aggregation (APA), which captures cross-object geometric consistency for each affordance, and Intra-Object Relational Modeling (IORM), which refines geometric differentiation within objects to support precise semantic alignment. We validate the effectiveness of our method through extensive experiments on a newly introduced benchmark, as well as two existing benchmarks, demonstrating superior performance in comparison with existing methods.
Paper Structure (60 sections, 24 equations, 9 figures, 10 tables)

This paper contains 60 sections, 24 equations, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. 3D Affordance Grounding
  4. Text-Point Cloud Cross-Modal Learning
  5. 3D Open Vocabulary Learning
  6. Method
  7. Overview
  8. Part-aware Instruction Representation
  9. Object-level Geometric Modeling
  10. Cross-modal Fusion Module
  11. Stage 1: Text-to-Point Fusion.
  12. Stage 2: Point-to-Text Fusion.
  13. Decoding and Affordance Mask Prediction
  14. Affordance Prototype Aggregation
  15. Training and Inference
  16. ...and 45 more sections

Figures (9)

  • Figure 1: Overview of motivations, illustrating (a) part-aware instruction generation and semantic–geometric alignment, (b) affordance prototype aggregation, (c) intra-object relational modeling.
  • Figure 2: Overview of our framework, comprising five stages: (1) Part-aware Instruction Representation (§\ref{['sec:PIGSGA']}); (2) Object-level Geometric Modeling (§\ref{['sec:PEIORM']}); (3) Cross-modal Fusion (§\ref{['sec:CMFM']}); (4) Decoding and Mask Prediction (§\ref{['sec:mask']}); and (5) Prototype Aggregation (§\ref{['sec:proto']}). Zoom-in views of IORM, CMFM, and MSSM are in the Appendix.
  • Figure 3: Visualization comparison. Results under four evaluation splits with predictions from each method. Red: predicted regions.
  • Figure 4: Cross-dataset zero-shot predictions showing semantically consistent results across domain shifts.
  • Figure S1: Model architecture details of three components: (a) Intra-Object Relational Modeling (IORM); (b) Cross-Modal Fusion Module (CMFM); (c) Multi-Scale Selection Module (MSSM).
  • ...and 4 more figures