Visual Affordance Prediction: Survey and Reproducibility

TL;DR

This work addresses the fragmented landscape of visual affordance prediction by proposing a unified problem formulation conditioned on the scene $x_v$, a task $\\mathcal{T}$, and a hand model $e$, outputting the target object $o$, action $a$, interaction region $S$, and hand pose $P$. It decomposes visual affordance into four core subtasks—object localisation, functional classification, functional segmentation, and hand pose estimation—with discussions of related methods, datasets, and their limitations. A key contribution is a thorough reproducibility analysis that identifies benchmarks, implementation availability, and setup details as critical gaps, and the introduction of Affordance Sheets to promote transparency and fair comparisons. The paper also outlines future directions, including multimodal integration, scalable datasets, and formal benchmarking protocols, to advance reliable, task-conditioned affordance prediction in real-world robotic and human–machine contexts.

Abstract

Affordances are the potential actions an agent can perform on an object, as observed by a camera. Visual affordance prediction is formulated differently for tasks such as grasping detection, affordance classification, affordance segmentation, and hand pose estimation. This diversity in formulations leads to inconsistent definitions that prevent fair comparisons between methods. In this paper, we propose a unified formulation of visual affordance prediction by accounting for the complete information on the objects of interest and the interaction of the agent with the objects to accomplish a task. This unified formulation allows us to comprehensively and systematically review disparate visual affordance works, highlighting strengths and limitations of both methods and datasets. We also discuss reproducibility issues, such as the unavailability of methods implementation and experimental setups details, making benchmarks for visual affordance prediction unfair and unreliable. To favour transparency, we introduce the Affordance Sheet, a document that details the solution, datasets, and validation of a method, supporting future reproducibility and fairness in the community.

Figures (3)

• Figure 1: Visual affordance prediction in case of a knife: what actions the agent performs, where the hand interacts with the object (heat map), and how the interaction is performed (hand pose for cutting). Legend: grasp, slide, cut, pierce.
• Figure 2: Comparison of topics discussed in affordance surveys: Hassanin et al. hassanin2021visual, Chen et al. chen2023survey, Ardón et al. ardon2020affordances. We propose an affordance formulation unifying previous redefinitions, we discuss the reproducibility issues and the limitations of datasets annotation preventing fair comparison across methods.
• Figure 3: Visualisation of datasets size based on number of images, number of affordance categories, and number of object categories. KEY: Affordance classification, Affordance detection and segmentation, Affordance grounding, Hand-object pose estimation.