Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning

TL;DR

This work presents SE(3)-equivariant Equivariant Descriptor Fields (EDFs) for end-to-end visual robotic manipulation from unsegmented point clouds, achieving high sample efficiency (training from a handful of demonstrations) and strong generalization to unseen poses, instances, and distractors. It couples SE(3) representation theory with a bi-equivariant energy-based model, where EDFs provide orientation-aware descriptors and a bi-equivariant energy encourages correct placement regardless of scene and grasp posture changes. The approach is implemented with Tensor Field Networks and SE(3)-Transformers, using SE(3)-equivariant query densities and MCMC-based sampling (MH on SE(3) followed by Langevin dynamics) to optimize the energy. Experiments on 6-DoF tasks demonstrate superior generalization and end-to-end performance compared to SE(3)-Transporter Networks and ablations, highlighting the importance of higher-type equivariant descriptors for orientation-sensitive manipulation. The work points to future directions in faster sampling and trajectory-level manipulation, expanding the practical reach of SE(3)-equivariant robotics.

Abstract

End-to-end learning for visual robotic manipulation is known to suffer from sample inefficiency, requiring large numbers of demonstrations. The spatial roto-translation equivariance, or the SE(3)-equivariance can be exploited to improve the sample efficiency for learning robotic manipulation. In this paper, we present SE(3)-equivariant models for visual robotic manipulation from point clouds that can be trained fully end-to-end. By utilizing the representation theory of the Lie group, we construct novel SE(3)-equivariant energy-based models that allow highly sample efficient end-to-end learning. We show that our models can learn from scratch without prior knowledge and yet are highly sample efficient (5~10 demonstrations are enough). Furthermore, we show that our models can generalize to tasks with (i) previously unseen target object poses, (ii) previously unseen target object instances of the category, and (iii) previously unseen visual distractors. We experiment with 6-DoF robotic manipulation tasks to validate our models' sample efficiency and generalizability. Codes are available at: https://github.com/tomato1mule/edf

Figures (14)

• Figure 1: Given few (5$\sim$10) demonstrations of a mug pick-and-place task, EDFs can be trained fully end-to-end without requiring any pre-training, object segmentation, or pose estimation pipelines. In addition, we show that EDFs can generalize to A) unseen poses, B) unseen instances of the target object category, and C) the presence of unseen visual distractors.
• Figure 2: A) The model is globally equivariant if the grasp pose is equivariant to the transformations of the whole scene (the target object and background). B) The model is locally equivariant to the target object if the grasp pose is equivariant to the localized transformations of the target object.
• Figure 3: A) Query points and query EDF are generated from the point cloud of the grasp. Query EDF values at the query points are used as the query descriptors. We visualized three type-$0$ descriptors in colors (RGB) and type-$1$ descriptors as arrows. We only visualized type-$1$ descriptors in important locations. We did not visualize higher-type descriptors. B) The key descriptors are generated from the point cloud of the scene. C) The query descriptors are transformed and matched to the key descriptors to produce the energy of the pose. For simplicity, we only visualized the query descriptor for a single query point. Note that the query and key descriptors are better aligned in the low energy case than in the high energy case for both the type-$0$ and type-$1$ descriptors (The orange query points are near the orange region, and the black arrow is well aligned to the gray arrows).
• Figure 4: The key EDF of a trained pick-model is illustrated for the scenes with a mug in A) upright pose and B) lying pose. Note that the colors (type-$0$ descriptors) are invariant to the rotation of the mug. On the other hand, the arrows (type-$1$ descriptors) are equivariant to the rotation. We only visualized type-$1$ descriptors in important locations. Higher-type descriptors are not visualized.
• Figure 5: A) Only ten demonstrations with objects in upright poses are provided during the training. B) The models are evaluated with unseen object instances in unseen poses with unseen distractors.

Theorems & Definitions (30)

• Definition 1
• Definition 2
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Proposition 5
• Proposition 6
• Proposition 7
• Proposition 8