SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulation

TL;DR

SPECI addresses the challenge of lifelong robot manipulation by introducing a three-tier hierarchical continual imitation learning framework that jointly learns perceptual fusion, dynamic skill inference, and action execution. It eliminates manual skill definitions through an expandable skill codebook and employs attention-driven Skill selection and Mode Approximation to enable efficient cross-task transfer at both skill and task levels. The approach combines a multimodal perception module, a high-level skill inference module, and a low-level GMM-based policy, enhanced by CP-based mode decomposition to separate shared and task-specific knowledge. Empirical results on the LIBERO benchmark demonstrate superior forward and backward knowledge transfer, reduced forgetting, and strong overall performance across diverse, long-horizon tasks, highlighting SPECI’s practical impact for scalable, autonomous robotic manipulation.

Abstract

Real-world robot manipulation in dynamic unstructured environments requires lifelong adaptability to evolving objects, scenes and tasks. Traditional imitation learning relies on static training paradigms, which are ill-suited for lifelong adaptation. Although Continual Imitation Learnin (CIL) enables incremental task adaptation while preserving learned knowledge, current CIL methods primarily overlook the intrinsic skill characteristics of robot manipulation or depend on manually defined and rigid skills, leading to suboptimal cross-task knowledge transfer. To address these issues, we propose Skill Prompts-based HiErarchical Continual Imitation Learning (SPECI), a novel end-to-end hierarchical CIL policy architecture for robot manipulation. The SPECI framework consists of a multimodal perception and fusion module for heterogeneous sensory information encoding, a high-level skill inference module for dynamic skill extraction and selection, and a low-level action execution module for precise action generation. To enable efficient knowledge transfer on both skill and task levels, SPECI performs continual implicit skill acquisition and reuse via an expandable skill codebook and an attention-driven skill selection mechanism. Furthermore, we introduce mode approximation to augment the last two modules with task-specific and task-sharing parameters, thereby enhancing task-level knowledge transfer. Extensive experiments on diverse manipulation task suites demonstrate that SPECI consistently outperforms state-of-the-art CIL methods across all evaluated metrics, revealing exceptional bidirectional knowledge transfer and superior overall performance.

Figures (5)

• Figure 1: (a) Overview of SPECI, which consists of three hierarchical modules: Multimodal Perception, Skill Inference and Action Execution. (b) Illustration of four CL scenarios, each demanding distinct knowledge transfer capabilities. Goal states are visualized for all tasks except the Different Layouts task suite, where initial states are displayed instead.
• Figure 2: (a) Framework of the proposed SPECI for robot continual imitation learning. The SPECI architecture consists of three hierarchical levels. The multimodal perception and fusion module encodes task descriptions, images of workspace and wrist views, and the robot’s proprioceptive state. In the high-level skill inference module, skill vectors are dynamically selected and weighted based on the output state embedding $\boldsymbol{s^{k,e}_t}$. The low-level action execution module then decodes and samples action $\boldsymbol{a^k_t}$ conditioned on the latent skill variable $\boldsymbol{z_t}$. Additionally, we enhance the transformer decoder in the last two levels with Mode Approximation to improve knowledge sharing and isolation across tasks. (b) Illustration of the skill selection and mode approximation mechanism. For the $k$-th task, SPECI first initializes a new skill subset for the current task, along with the corresponding skill keys, attention vectors and task-specific decomposition vector $\boldsymbol{Q^k}$. Then, the synthesized latent skill $\boldsymbol{\Tilde{p}}$ is computed via a weighted summation according to the state information and is subsequently divided into $\boldsymbol{p_K}$ and $\boldsymbol{p_V}$. Finally, these components together with $\boldsymbol{Q^k}$ are input into the transformer decoder blocks to enhance knowledge transfer on both skill and task levels.
• Figure 3: Comparison of different policy architectures under ER chaudhry2019tiny and PACKNET mallya2018packnet lifelong learning paradigms, evaluated on LIBERO-OBJECT and LIBERO-GOAL task suites. We report the success rate for each task throughout the whole continual learning procedure. Each Task i subplot illustrates the agent's performance on the $i$-th task after training on the corresponding task ($x$-axis).
• Figure 4: Visualization of skill vector cross-task reuse in our SPECI under PackNet paradigm after learning task 4 on LIBERO-GOAL. The top-10 most frequently selected skills are displayed with color indicating their source skill subset and numerical labels showing their original indices.
• Figure 5: Visualization of the FWT and AUC metric gaps between the upper bounds (wide solid bars) and different policy architectures under PackNet paradigm (narrow hatched bars), evaluated across four task suites.