Closed-Loop Transfer for Weakly-supervised Affordance Grounding

TL;DR

LoopTrans tackles weakly-supervised affordance grounding by establishing a closed-loop knowledge transfer between exocentric interaction views and egocentric object views. It introduces a unified shared CAM to jointly activate affordances across views, a region-to-pixel localization pathway with object-part pseudo-masks, and an ego-to-exo denoising distillation to suppress background noise and refine activations. The approach yields consistent improvements on image and video benchmarks, including challenging occluded interactions, and demonstrates robust cross-domain generalization. This framework advances practical affordance grounding by reducing domain gaps and enabling feedback-driven refinement of both activation and localization stages, with potential impact on embodied AI tasks and human-object interaction understanding.

Abstract

Humans can perform previously unexperienced interactions with novel objects simply by observing others engage with them. Weakly-supervised affordance grounding mimics this process by learning to locate object regions that enable actions on egocentric images, using exocentric interaction images with image-level annotations. However, extracting affordance knowledge solely from exocentric images and transferring it one-way to egocentric images limits the applicability of previous works in complex interaction scenarios. Instead, this study introduces LoopTrans, a novel closed-loop framework that not only transfers knowledge from exocentric to egocentric but also transfers back to enhance exocentric knowledge extraction. Within LoopTrans, several innovative mechanisms are introduced, including unified cross-modal localization and denoising knowledge distillation, to bridge domain gaps between object-centered egocentric and interaction-centered exocentric images while enhancing knowledge transfer. Experiments show that LoopTrans achieves consistent improvements across all metrics on image and video benchmarks, even handling challenging scenarios where object interaction regions are fully occluded by the human body.

Figures (9)

• Figure 1: Visualization samples for lie on and ride affordances on egocentric images, comparing state-of-the-art methods with ours.
• Figure 3: Overall framework of our proposed LoopTrans enables closed affordance knowledge transfer, with the process interaction → activation → localization → activation, spanning both exocentric and egocentric domains.
• Figure 4: (a) Network architectures of $\varTheta_{\text{SCAM}}$ and our proposed $\varTheta_{\text{SCAM}}^{\text{+noise}}$; (b) The mechanism of our denoising distillation; (c) Exocentric activation results of$\varTheta_{\text{SCAM}}$ and $\varTheta_{\text{SCAM}}^{\text{+noise}}$.
• Figure 5: Visualization results compared with existing methods.
• Figure 6: Comparison of knowledge activation on exocentric images between our LoopTrans and the previous one-way transfer framework.