DeCo: Task Decomposition and Skill Composition for Zero-Shot Generalization in Long-Horizon 3D Manipulation

TL;DR

DeCo introduces a physically grounded, model-agnostic framework to achieve zero-shot generalization for compositional long-horizon 3D manipulation by decomposing demonstrations into atomic skills based on gripper-object interactions, learning those skills, and leveraging vision-language models for planning. A spatially-aware skill chaining module ensures collision-free transitions between skills during execution. The authors validate DeCo on the DeCoBench simulation benchmark and in real-world experiments across multiple baseline IL models, demonstrating substantial gains in long-horizon task success and generalization to unseen compositions. The work highlights the importance of aligning semantic plans with executable skills and provides a practical pathway for scalable, general-purpose robotic manipulation in unstructured environments.

Abstract

Generalizing language-conditioned multi-task imitation learning (IL) models to novel long-horizon 3D manipulation tasks remains a significant challenge. To address this, we propose DeCo (Task Decomposition and Skill Composition), a model-agnostic framework compatible with various multi-task IL models, designed to enhance their zero-shot generalization to novel, compositional, long-horizon 3D manipulation tasks. DeCo first decomposes IL demonstrations into a set of modular atomic tasks based on the physical interaction between the gripper and objects, and constructs an atomic training dataset that enables models to learn a diverse set of reusable atomic skills during imitation learning. At inference time, DeCo leverages a vision-language model (VLM) to parse high-level instructions for novel long-horizon tasks, retrieve the relevant atomic skills, and dynamically schedule their execution; a spatially-aware skill-chaining module then ensures smooth, collision-free transitions between sequential skills. We evaluate DeCo in simulation using DeCoBench, a benchmark specifically designed to assess zero-shot generalization of multi-task IL models in compositional long-horizon 3D manipulation. Across three representative multi-task IL models (RVT-2, 3DDA, and ARP), DeCo achieves success rate improvements of 66.67%, 21.53%, and 57.92%, respectively, on 12 novel compositional tasks. Moreover, in real-world experiments, a DeCo-enhanced model trained on only 6 atomic tasks successfully completes 9 novel long-horizon tasks, yielding an average success rate improvement of 53.33% over the base multi-task IL model. Video demonstrations are available at: https://deco226.github.io.

Figures (11)

• Figure 1: We present DeCo, a model-agnostic framework that enables diverse multi-task IL models to zero-shot generalize to novel yet compositional long-horizon 3D manipulation tasks.
• Figure 2: An overview of DeCo framework.
• Figure 3: Visual example of full and half interactions.
• Figure 4: Ablation study of heuristic settings in DeCo. (b) and (c) are based on the RVT-2+DeCo model.
• Figure 5: Two visual failure cases of 3DDA+DeCo and ARP+DeCo.