FSAG: Enhancing Human-to-Dexterous-Hand Finger-Specific Affordance Grounding via Diffusion Models

TL;DR

FSAG addresses dexterous grasp synthesis without relying on large robot grasp datasets by grounding finger-specific affordances in latent semantics from pretrained diffusion models. It ground per-finger contact likelihoods using diffusion-derived hyperfeatures, decodes them into dense heatmaps, and maps them to geometry-aligned approach vectors for a kinematics-aware planner that transfers across dexterous hands. The approach yields state-of-the-art grounding metrics and high grasp success on seen and unseen objects, while enabling cross-embodiment generalization with minimal retraining. Practically, FSAG demonstrates a scalable, demonstration-light pathway to dexterous manipulation driven by foundation-model semantics and object-centric priors. Occasional failures on slender objects point to future work on residual impedance control and contact-force estimation to improve robustness.

Abstract

Dexterous grasp synthesis remains a central challenge: the high dimensionality and kinematic diversity of multi-fingered hands prevent direct transfer of algorithms developed for parallel-jaw grippers. Existing approaches typically depend on large, hardware-specific grasp datasets collected in simulation or through costly real-world trials, hindering scalability as new dexterous hand designs emerge. To this end, we propose a data-efficient framework, which is designed to bypass robot grasp data collection by exploiting the rich, object-centric semantic priors latent in pretrained generative diffusion models. Temporally aligned and fine-grained grasp affordances are extracted from raw human video demonstrations and fused with 3D scene geometry from depth images to infer semantically grounded contact targets. A kinematics-aware retargeting module then maps these affordance representations to diverse dexterous hands without per-hand retraining. The resulting system produces stable, functionally appropriate multi-contact grasps that remain reliably successful across common objects and tools, while exhibiting strong generalization across previously unseen object instances within a category, pose variations, and multiple hand embodiments. This work (i) introduces a semantic affordance extraction pipeline leveraging vision-language generative priors for dexterous grasping, (ii) demonstrates cross-hand generalization without constructing hardware-specific grasp datasets, and (iii) establishes that a single depth modality suffices for high-performance grasp synthesis when coupled with foundation-model semantics. Our results highlight a path toward scalable, hardware-agnostic dexterous manipulation driven by human demonstrations and pretrained generative models.

Figures (6)

• Figure 2: Pipeline overview. (1) Hyperfeature extraction: A frozen text-to-image diffusion U-Net encodes the object image with text conditioning. Multi-timestep, multi-scale activations are aggregated into hyperfeatures $A_g$. (2) Finger-specific affordance grounding: An FPN-style decoder maps $A_g$ to five per-finger likelihood maps $\hat{H}$ supervised by fingertip labels from human demonstrations. (3) Manipulation: Back-projecting GroundingDINO–SAM2 masks with FoundationStereo depth yields a partial object point cloud; peaks of $\hat{H}$ are lifted to 3D and local normals define phase-labeled waypoints (approach, closure, hold).A damped least-squares QP tracks these waypoints under joint and collision constraints to execute dexterous grasps.
• Figure 3: Right: (1) finger-specific affordance likelihoods predicted from diffusion-derived hyperfeatures; (2) partial object point cloud reconstructed by back-projecting GroundingDINO–SAM2 segmentation with FoundationStereo depth; (3) approach vectors obtained from local surface normals around the selected contact candidates. Left: a damped least-squares QP tracks phase-labeled waypoints to execute a dexterous grasp.
• Figure 4: Robotic arm and dexterous-hand experimental platform used in our study.
• Figure 5: Qualitative comparison of finger-specific affordance grounding. Left: Seen objects; Right: Unseen objects. Rows show ReKep, CMKA, Ours (DINO features), and Ours (Stable Diffusion). Overlays visualize five per-finger likelihood maps; colored dots indicate annotated fingertip contacts. The diffusion-based variant produces sharper, finger-disentangled hotspots aligned with functional parts and preserves localization quality on unseen tools. ReKep and CMKA each produce keypoints directly and a single-channel heatmap is generated via post-processing following CMKA, thus their overlays are rendered in a single purple color, whereas our method outputs five-channel heatmaps rendered with five distinct colors.
• Figure 6: Affordance-driven dexterous grasping in the real world. Representative rollouts on everyday objects and tools; columns denote tasks (banana, bottle, tomato, screwdriver, drill).