Skill-Aware Diffusion for Generalizable Robotic Manipulation

TL;DR

SADiff tackles generalization in robotic manipulation by explicitly modeling skill-level information. It introduces a three-part pipeline: a skill-aware encoder with learnable tokens to condition multimodal inputs, a skill-constrained diffusion model to generate object-centric 2D motion flow, and a skill-retrieval transformation that uses trajectory priors to map flows into executable 3D actions. The authors also present IsaacSkill, a high-fidelity dataset for skill-centric evaluation, enabling robust sim-to-real transfer. Across simulation and real-world experiments, SADiff demonstrates superior generalization to unseen objects, environments, and embodiments, and even zero-shot transfer without real-world fine-tuning, highlighting the practical impact of integrating skill priors throughout the manipulation pipeline.

Abstract

Robust generalization in robotic manipulation is crucial for robots to adapt flexibly to diverse environments. Existing methods usually improve generalization by scaling data and networks, but model tasks independently and overlook skill-level information. Observing that tasks within the same skill share similar motion patterns, we propose Skill-Aware Diffusion (SADiff), which explicitly incorporates skill-level information to improve generalization. SADiff learns skill-specific representations through a skill-aware encoding module with learnable skill tokens, and conditions a skill-constrained diffusion model to generate object-centric motion flow. A skill-retrieval transformation strategy further exploits skill-specific trajectory priors to refine the mapping from 2D motion flow to executable 3D actions. Furthermore, we introduce IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer. Experiments in simulation and real-world settings show that SADiff achieves good performance and generalization across various manipulation tasks. Code, data, and videos are available at https://sites.google.com/view/sa-diff.

Figures (11)

• Figure 1: Overview of the proposed Skill-Aware Diffusion (SADiff) framework. It is structured into three phases: (1) The encoding phase, where the skill-aware encoding module uses learnable skill tokens to interact with multimodal inputs and extract skill-specific information; (2) The generation phase, in which a skill-constrained diffusion model generates object-centric motion flow conditioned on the skill-aware token sequences, optimized by both denoising loss and two skill-specific auxiliary losses; and (3) The execution phase, which employs a skill-retrieval transformation strategy to translate the generated 2D motion flow into executable 3D trajectories by leveraging skill-specific priors.
• Figure 2: Architecture of the skill-aware encoding module. The skill-aware encoding module integrates image, language, and bounding boxes of relevant objects with learnable skill tokens through attention-based interactions, producing skill-aware token sequences.
• Figure 3: Overview of the skill-constrained diffusion model. The diffusion model generates motion flow by jointly optimizing skill classification loss, skill contrastive loss, and denoising loss to ensure accurate skill selection, semantic alignment, and precise flow generation.
• Figure 4: Overview of the constructed IsaacSkill dataset. The dataset comprises five fundamental manipulation skills: "Pouring", "Picking&Placing", "Pushing", "Slide Opening", and "Hinge Opening". Each skill includes three different tasks involving different objects.
• Figure 5: Overview of the generalization evaluation settings. Left: The original collected demonstration. Right: The model is evaluated under four different types of variations involving distribution shifts in backgrounds, intra-category instances, cross-category objects, and embodiment.