Learning Diffusion Policy from Primitive Skills for Robot Manipulation

TL;DR

This work tackles misalignment between high-level task instructions and fine-grained robot actions by introducing SDP, a skill-conditioned diffusion policy. SDP explicitly learns eight primitive skills, uses a vision-language model to map observations and language to discrete skill representations, and employs a lightweight router to assign a skill per state, producing a single-skill diffusion policy that generates skill-aligned actions. The approach achieves state-of-the-art performance on CALVIN and LIBERO benchmarks and demonstrates strong real-world generalization, interpretability through explicit skills, and robustness to visual variation. The results suggest a practical paradigm for integrating interpretable primitive skills with diffusion-based control to enable reliable multi-task manipulation.

Abstract

Diffusion policies (DP) have recently shown great promise for generating actions in robotic manipulation. However, existing approaches often rely on global instructions to produce short-term control signals, which can result in misalignment in action generation. We conjecture that the primitive skills, referred to as fine-grained, short-horizon manipulations, such as ``move up'' and ``open the gripper'', provide a more intuitive and effective interface for robot learning. To bridge this gap, we propose SDP, a skill-conditioned DP that integrates interpretable skill learning with conditional action planning. SDP abstracts eight reusable primitive skills across tasks and employs a vision-language model to extract discrete representations from visual observations and language instructions. Based on them, a lightweight router network is designed to assign a desired primitive skill for each state, which helps construct a single-skill policy to generate skill-aligned actions. By decomposing complex tasks into a sequence of primitive skills and selecting a single-skill policy, SDP ensures skill-consistent behavior across diverse tasks. Extensive experiments on two challenging simulation benchmarks and real-world robot deployments demonstrate that SDP consistently outperforms SOTA methods, providing a new paradigm for skill-based robot learning with diffusion policies.

Figures (5)

• Figure 1: A task consists of a series of short-term manipulations, and we abstract them into eight shared primitive skills, which provide concrete instruction. Previous diffusion policies map high-level instructions to actions directly. In contrast, we learn primitive skills and integrate them into the conditional action generation for more precise control.
• Figure 2: Comparison between (a) diffusion policy (DP), (b) language-conditioned DP, and (c) our skill-conditioned DP. Ours executes abstract instructions with more precise guidance from the assigned primitive skills.
• Figure 3: Overview of the proposed skill-conditioned diffusion policy (SDP). SDP abstracts short-term manipulations across different tasks into eight primitive skills and introduces a unified prompt template to specify the upcoming manipulations. A router is then designed to assign importance scores for all candidate skills based on the embedding $\bm{z}_{vl}$, generated from visual observations and language instructions by a VLM. Furthermore, a skill with the highest score is selected, which parameterizes an additional FFN layer in the diffusion policy. Other information, such as proprioception, is encoded by an MLP and further injected via an AdaLN operation, resulting in a single-skill policy that predicts skill-aligned actions for precise control.
• Figure 4: Task success rates (%) on real-world robot manipulation tasks. We specially designed 9 tasks (see the right figure) to evaluate two aspects of policy ability: multi-task learning (the first six tasks) and visual generalization (the last three tasks). In the visual generalization setting, we further investigate the generalization to unseen objects (an apple and a banana) and the robustness to visual distractors. The proposed SDP (pink) consistently outperforms baselines (green and orange), demonstrating better generalization across tasks and objects as well as robustness to distractors.
• Figure 5: Visualizations on assigned skills. The left plots draw assigned skills at each timestep, where the horizontal axis denotes the timestep, and the vertical axis corresponds to different primitive skills. Images on the right correspond to the observations by performing the skills. SDP learns primitive skills during training and composes them to accomplish complex tasks in inference.