Learning Generalizable 3D Manipulation With 10 Demonstrations

TL;DR

This work presents a novel framework that learns manipulation skills from as few as 10 demonstrations, yet still generalizes to spatial variants such as different initial object positions and camera viewpoints, and introduces a critical spatially equivariant training strategy that captures the spatial knowledge embedded in expert demonstrations.

Abstract

Learning robust and generalizable manipulation skills from demonstrations remains a key challenge in robotics, with broad applications in industrial automation and service robotics. While recent imitation learning methods have achieved impressive results, they often require large amounts of demonstration data and struggle to generalize across different spatial variants. In this work, we present a novel framework that learns manipulation skills from as few as 10 demonstrations, yet still generalizes to spatial variants such as different initial object positions and camera viewpoints. Our framework consists of two key modules: Semantic Guided Perception (SGP), which constructs task-focused, spatially aware 3D point cloud representations from RGB-D inputs; and Spatial Generalized Decision (SGD), an efficient diffusion-based decision-making module that generates actions via denoising. To effectively learn generalization ability from limited data, we introduce a critical spatially equivariant training strategy that captures the spatial knowledge embedded in expert demonstrations. We validate our framework through extensive experiments on both simulation benchmarks and real-world robotic systems. Our method demonstrates a 60 percent improvement in success rates over state-of-the-art approaches on a series of challenging tasks, even with substantial variations in object poses and camera viewpoints. This work shows significant potential for advancing efficient, generalizable manipulation skill learning in real-world applications.

Figures (5)

• Figure 1: In a), we report the average success rates of the two methods on a series of challenging tasks. In b), we show the average success rates of the two methods under different camera viewpoint settings. In c), we progressively expand the random initialization region where the manipulated object is located at the start of the tasks. Compared to the state-of-the-art 3D manipulation learning method DP3, our framework demonstrates significant improvements in success rates and generalization ability.
• Figure 2: Overall workflow of our developed framework, which consists of two key modules: Semantic Guided Perception (SGP) and Spatial Generalized Decision (SGD). In (a) and (b), SGP takes the RGB-D images $\mathbf{I}$ and gripper pose $\mathbf{h}$ as inputs, constructing a 3D point cloud representation $\mathbf{p}$ that is both task-focused and spatially aware. The details of SGP are shown in (c). SGD is an efficient decision-making module based on diffusion policy. To enable our framework to generalize across different 3D spatial variations, we introduce a spatially equivariant training strategy that leverages the spatial knowledge embedded in expert trajectories, as illustrated in (d).
• Figure 3: We compared two choices of coordinate reference. When using the robot base as the reference, the perceived points of the manipulated object vary significantly between trajectories due to differences in initial and target poses. In contrast, when using the gripper base as the reference, the points of the manipulated objects across different trajectories converge to the same state, as the relative pose between the gripper and the objects remains consistent.
• Figure 4: Given the original point cloud $\mathbf{p}$, gripper pose $\mathbf{h}$, action $\mathbf{a}$, we apply a random rotation $\mathbf{T}$ to get the augmented data $\mathbf{T}(\mathbf{p})$, $\mathbf{T}(\mathbf{h})$, and corresponding $\mathbf{T}(\mathbf{a}).$
• Figure 5: Here we visualize the different camera viewpoint settings used in evaluation. We apply movement along each of the six degrees of freedom, including three translations and three rotations, resulting in a total of six distinct settings.