AFFORD2ACT: Affordance-Guided Automatic Keypoint Selection for Generalizable and Lightweight Robotic Manipulation

TL;DR

AFFORD2ACT tackles the challenge of learning manipulation policies from dense visual inputs by distilling a compact, semantically grounded set of 2D keypoints guided by text-prompted affordances. A three-stage pipeline localizes affordance regions, constructs a small keypoint pool, and learns a transformer-based policy with gating to emphasize task-relevant points, yielding a $38$-dimensional state and training in about $15$ minutes. Across six real-world tasks, the method achieves strong data efficiency and robust generalization to unseen objects, backgrounds, and distractors, with an $82\%$ success rate on unseen instances and open-vocabulary prompts. These results highlight the practical impact of combining affordance grounding with keypoint distillation to enable fast, scalable, and generalizable robotic manipulation without heavy 3D perception or manual annotation.

Abstract

Vision-based robot learning often relies on dense image or point-cloud inputs, which are computationally heavy and entangle irrelevant background features. Existing keypoint-based approaches can focus on manipulation-centric features and be lightweight, but either depend on manual heuristics or task-coupled selection, limiting scalability and semantic understanding. To address this, we propose AFFORD2ACT, an affordance-guided framework that distills a minimal set of semantic 2D keypoints from a text prompt and a single image. AFFORD2ACT follows a three-stage pipeline: affordance filtering, category-level keypoint construction, and transformer-based policy learning with embedded gating to reason about the most relevant keypoints, yielding a compact 38-dimensional state policy that can be trained in 15 minutes, which performs well in real-time without proprioception or dense representations. Across diverse real-world manipulation tasks, AFFORD2ACT consistently improves data efficiency, achieving an 82% success rate on unseen objects, novel categories, backgrounds, and distractors.

Figures (11)

• Figure 1: Top: policy trained with original prompts (“hold”, “stir”) executes the actions over time ($t$). Middle: at test time, the same policy generalizes to synonym prompts (“pick”, “mix”). Bottom: affordance-guided keypoints with learned importance (green; variable count; ring radius $\propto$ attention) remain stable under unseen object shapes, lighting conditions, dynamic distractors, and scene clutter—leading to successful rollouts.
• Figure 2: Overview of the Afford2Act pipeline. We extract task-relevant object keypoints using affordance masks and DINO filtering, track them across frames, and embed them with a transformer and attention gating. The policy head predicts pose and gripper commands from these compact keypoint representations, enabling vision-based manipulation without requiring proprioception.
• Figure 3: Real-world rollouts of Afford2Act on Pour (top) and Cut (bottom) tasks. In each sequence (left to right: approach $\rightarrow$ interaction $\rightarrow$ completion), green dots mark keypoints automatically selected on the affordance region (e.g., knife for "cut", handle/spout for "pour"). The translucent ring size reflects the attention weight at each step. While keypoints remain fixed on task-relevant parts, the policy dynamically shifts its focus: for "cut", the attention moves from the blade tip toward the handle; for "pour", from the edge to the main handle. This adaptive attention enables robust, context-aware manipulation.
• Figure 4: (Left) A figure of our physical setup, composing of a UR3e Robotic arm and one side RealSenseD435i Camera. (Right) shows the objects we used for seen (right) and unseen (left) scenarios.
• Figure 5: Qualitative generalization on two tasks. Each row shows rollouts for Pour (top) and Kick (bottom). Left: training objects (seen). Right: held-out settings with unseen scenes (new backgrounds/tabletops) and, for Pour, unseen object instances. Within each panel we show three key frames (setup, approach, completion). Without any test-time adaptation, Afford2Act trained from single-view RGB using affordance-guided keypoints successfully transfers by attending to the functional part (mug handle / ball) while ignoring clutter.