GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulation

TL;DR

This work tackles learning actionable affordances from human demonstrations to enable robust robotic manipulation. It introduces HOVA-500K, a large-scale, affordance-annotated dataset, and GLOVER++, a global-to-local framework that distills affordance knowledge into open-vocabulary reasoning using a vision-language backbone. The approach achieves state-of-the-art performance on HOVA-500K and generalizes to zero-shot manipulation, imitation learning, long-horizon planning, and bimanual tasks, across simulated and real environments. By making affordances explicit and transferable, the work bridges human demonstrations and robotic control with interpretable reasoning and practical applicability.

Abstract

Learning manipulation skills from human demonstration videos offers a promising path toward generalizable and interpretable robotic intelligence-particularly through the lens of actionable affordances. However, transferring such knowledge remains challenging due to: 1) a lack of large-scale datasets with precise affordance annotations, and 2) insufficient exploration of affordances in diverse manipulation contexts. To address these gaps, we introduce HOVA-500K, a large-scale, affordance-annotated dataset comprising 500,000 images across 1,726 object categories and 675 actions. We also release a standardized benchmarking suite for multi-modal affordance reasoning. Built upon HOVA-500K, we present GLOVER++, a global-to-local affordance training framework that effectively transfers actionable affordance knowledge from human demonstrations to downstream open-vocabulary reasoning tasks. GLOVER++ achieves state-of-the-art results on the HOVA-500K benchmark and demonstrates strong generalization across diverse downstream robotic manipulation tasks. By explicitly modeling actionable affordances, GLOVER++ facilitates robust transfer across scenes, modalities, and tasks. We hope that HOVA-500K and the GLOVER++ framework will serve as valuable resources for bridging the gap between human demonstrations and robotic manipulation capabilities.

Figures (20)

• Figure 1: (a) GLOVER++ aims to learn generalizable affordance representation from human behaviors (e.g.open drawer). (b) The training pipeline of GLOVER++. We adopt a global-to-local decoding policy to balance global semantic decoding and local affordance decoding. (c) GLOVER++ is capable of transferring affordable knowledge to all kinds of distributions (simulation, sketch, cartoonetc.) in an open-vocabulary manner. It also presents strong spatial reasoning ability as shown in the bottom line. (d) By lifting inferred affordable points into 3D space, GLOVER++ provides perceptive awareness for real-world manipulation tasks. (Red dots represent affordable points.)
• Figure 2: The distribution of primary action categories ($>$1,000 samples) and related objects in HOVA-500K.
• Figure 3: Visualization of the decoded features by the global and local decoder (the intensity of highlight scale with interest of regions). We can observe that the integration of the local decoder effectively eliminates the background noise from the global decoding.
• Figure 4: Comparison with Qwen-2.5-VL, GLOVER++ generates more physically plausible and functionally grounded prediction results, aligning better with real-world interaction constraints.
• Figure 5: Left: GLOVER++ serves as a perceptual module for the VLM planner to complete long-horizon tasks. Right: GLOVER++ enables bimanual tasks by reasoning affordances for both left and right hands with spatial relationships.