Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning

Abstract

A core problem of Embodied AI is to learn object manipulation from observation, as humans do. To achieve this, it is important to localize 3D object affordance areas through observation such as images (3D affordance grounding) and understand their functionalities (affordance classification). Previous attempts usually tackle these two tasks separately, leading to inconsistent predictions due to lacking proper modeling of their dependency. In addition, these methods typically only ground the incomplete affordance areas depicted in images, failing to predict the full potential affordance areas, and operate at a fixed scale, resulting in difficulty in coping with affordances significantly varying in scale with respect to the whole object. To address these issues, we propose a novel approach that learns an affordance-aware 3D representation and employs a stage-wise inference strategy leveraging the dependency between grounding and classification tasks. Specifically, we first develop a cross-modal 3D representation through efficient fusion and multi-scale geometric feature propagation, enabling inference of full potential affordance areas at a suitable regional scale. Moreover, we adopt a simple two-stage prediction mechanism, effectively coupling grounding and classification for better affordance understanding. Experiments demonstrate the effectiveness of our method, showing improved performance in both affordance grounding and classification.

Figures (12)

• Figure 1: Motivation of Our Method: a) The cascaded model effectively couples grounding and classification; b) The inherent geometric similarity among different parts of an object allows for deriving the full potential interactive areas from a specific interaction; c) Learning affordance areas from a single scale is inefficient due to the wide span of scales, while multi-scale geometric features facilitate effective learning of affordance areas across this range of scales.
• Figure 2: Overview of our framework. It identifies the affordance region and category in four steps: 1) Extract 2D context-aware affordance feature $\mathbf{F}_{I}$ and multi-scale 3D geometric features $\mathbf{F}_{P}^{l}$ and $\mathbf{F}_{P}^{s}$(Sec. \ref{['feature_extra']}); 2) Fuse the two modalities, obtain multiscale features $\mathbf{\tilde{F}}_P^l$, $\mathbf{\tilde{F}}_P^s$, $\mathbf{\tilde{F}}_I^l$ and $\mathbf{\tilde{F}}_I^s$ (Sec. \ref{['CMFM']}); 3) Propogate the regional features at each scale and conduct scale selection to generate final representations $\tilde{\mathbf{R}}_{P}$ and $\tilde{\mathbf{F}}_{I}$ (Sec. \ref{['GFPMSS']}); 4) Predict the probabilistic mask $\hat{\phi}$ and affordance category $\hat{y}$ (Sec. \ref{['final_decoder']}).
• Figure 3: The visualization results. The first row displays the interactive images that reflect object affordances. The last row shows the ground truth of the affordance regions in the 3D point cloud. The left four columns correspond to the Seen setting, while the right four columns represent the Unseen setting.
• Figure 4: Visualization of ablation studies: (a) Without GFPM, predictions fail to cover the full affordance area. (b) Removing MSI reduces precision for fine-grained regions. (c) Without CGC, grounding areas are confused with incorrect functionalities.
• Figure 5: Visualization of cross-dataset generalization results on ShapeNet point clouds