Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

23 DoF Grasping Policies from a Raw Point Cloud

Martin Matak, Karl Van Wyk, Tucker Hermans

TL;DR

This paper proposes a novel imitation learning approach to learn a policy that directly predicts 23 DoF grasp trajectories from a partial point cloud provided by a single, fixed camera, and shows that the policy is capable of generalizing to novel objects.

Abstract

Coordinating the motion of robots with high degrees of freedom (DoF) to grasp objects gives rise to many challenges. In this paper, we propose a novel imitation learning approach to learn a policy that directly predicts 23 DoF grasp trajectories from a partial point cloud provided by a single, fixed camera. At the core of the approach is a second-order geometric-based model of behavioral dynamics. This Neural Geometric Fabric (NGF) policy predicts accelerations directly in joint space. We show that our policy is capable of generalizing to novel objects, and combine our policy with a geometric fabric motion planner in a loop to generate stable grasping trajectories. We evaluate our approach on a set of three different objects, compare different policy structures, and run ablation studies to understand the importance of different object encodings for policy learning.

23 DoF Grasping Policies from a Raw Point Cloud

TL;DR

This paper proposes a novel imitation learning approach to learn a policy that directly predicts 23 DoF grasp trajectories from a partial point cloud provided by a single, fixed camera, and shows that the policy is capable of generalizing to novel objects.

Abstract

Coordinating the motion of robots with high degrees of freedom (DoF) to grasp objects gives rise to many challenges. In this paper, we propose a novel imitation learning approach to learn a policy that directly predicts 23 DoF grasp trajectories from a partial point cloud provided by a single, fixed camera. At the core of the approach is a second-order geometric-based model of behavioral dynamics. This Neural Geometric Fabric (NGF) policy predicts accelerations directly in joint space. We show that our policy is capable of generalizing to novel objects, and combine our policy with a geometric fabric motion planner in a loop to generate stable grasping trajectories. We evaluate our approach on a set of three different objects, compare different policy structures, and run ablation studies to understand the importance of different object encodings for policy learning.

Paper Structure

This paper contains 13 sections, 6 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Imitation Learning Setup
  3. Data Collection
  4. Surrogate Expert and Training
  5. Neural Policy Architectures
  6. Geometric Fabrics
  7. Neural Geometric Fabric
  8. Acceleration MLP
  9. Object encoding
  10. Acceleration Integrator
  11. Motion Planner In The Loop
  12. Experiments
  13. Conclusion

Figures (4)

  • Figure 2: Region which we sample from above the object is computed based on the object's bounding box. Left: Training data is collected by generating trajectories from the grasp configuration to the sampled poses and then reversing those trajectories. Right: At evaluation time, we sample a target pose above the object and use a motion planner to get there. Then, the policy predicts a trajectory to grasp the object.
  • Figure 3: Example point cloud reconstruction; input in blue, output in red. While imperfect (left), it preserves information about object pose and course geometry (right).
  • Figure 4: Top: The red lines show the camera location and workspace boundaries. The objects are (left to right): 'mustard', 'sugarbox', 'bleach'. Bottom: Our manipulation pipeline from left to right: a motion planner moves the robot close to the object, our policy grasps the object, and then a motion planner lifts the object.
  • Figure 5: Grasp success rates for different grasp-trajectory predicting models and the dataset we use for training. Not all samples in the dataset are successful grasps. Input modalities: POS-position only, PCD - pointcloud.