H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows

TL;DR

H2OFlow tackles the challenge of learning comprehensive 3D human–object affordances by moving beyond contact-only cues to include orientation and spatial occupancy. The method synthetically generates diverse HOI samples with a 3D generative model, then learns a dense diffused flow representation via a diffusion transformer to predict multimodal human-object configurations from object point clouds. Affordances are inferred as probabilistic scores over pairings of human and object points, using a diffusion-based latent space to capture multi-modality. Empirical results show strong generalization to unseen real objects, improved accuracy over mesh- and annotation-dependent baselines, and robust performance under occlusion, with broader implications for robotics and AI agents interacting with unstructured environments.

Abstract

Understanding how humans interact with the surrounding environment, and specifically reasoning about object interactions and affordances, is a critical challenge in computer vision, robotics, and AI. Current approaches often depend on labor-intensive, hand-labeled datasets capturing real-world or simulated human-object interaction (HOI) tasks, which are costly and time-consuming to produce. Furthermore, most existing methods for 3D affordance understanding are limited to contact-based analysis, neglecting other essential aspects of human-object interactions, such as orientation (\eg, humans might have a preferential orientation with respect certain objects, such as a TV) and spatial occupancy (\eg, humans are more likely to occupy certain regions around an object, like the front of a microwave rather than its back). To address these limitations, we introduce \emph{H2OFlow}, a novel framework that comprehensively learns 3D HOI affordances -- encompassing contact, orientation, and spatial occupancy -- using only synthetic data generated from 3D generative models. H2OFlow employs a dense 3D-flow-based representation, learned through a dense diffusion process operating on point clouds. This learned flow enables the discovery of rich 3D affordances without the need for human annotations. Through extensive quantitative and qualitative evaluations, we demonstrate that H2OFlow generalizes effectively to real-world objects and surpasses prior methods that rely on manual annotations or mesh-based representations in modeling 3D affordance.

Figures (8)

• Figure 1: H2OFlow learns comprehensive affordances from synthetic 3D HOI data generated by 3D generative models using a novel representation. The learned affordance captures contact, orientational, and occupancy information based on input object point clouds.
• Figure 2: Visual illustration of affordance inference. Given predicted human point clouds, contact affordance assigns high scores to human-object point pairs that are close. Orientational affordances give higher scores to point pairs that yield more uniform cross-product directions (i.e., hand points) and vice versa (i.e., foot points). The spatial affordances output higher scores to regions surrounding the object that are often occupied by human parts. A video of the figure is available at this https://sites.google.com/view/h2oflow/home.
• Figure 3: Visualizations of some affordances inferred from dense diffused flows prediction. H2OFlow infers diverse affordance distributions from predicted HOI samples on unseen objects.
• Figure 4: (a) Ablations on cross-attention weights and (b) results on real-world point clouds. Objects shown are: monitor, trashcan, backpack handle & panel, chair, yoga ball, table, box, and suitcase.
• Figure 5: H2OFlow overview. Note the "Object" affordance matrix here is the transpose of the human contact affordance matrix.