DiffusionSeeder: Seeding Motion Optimization with Diffusion for Rapid Motion Planning

TL;DR

DiffusionSeeder is proposed, a diffusion based approach that generates trajectories to seed motion optimization for rapid robot motion planning that takes the initial depth image observation of the scene and generates high quality, multi-modal trajectories that are then fine-tuned with a few iterations of motion optimization.

Abstract

Running optimization across many parallel seeds leveraging GPU compute have relaxed the need for a good initialization, but this can fail if the problem is highly non-convex as all seeds could get stuck in local minima. One such setting is collision-free motion optimization for robot manipulation, where optimization converges quickly on easy problems but struggle in obstacle dense environments (e.g., a cluttered cabinet or table). In these situations, graph-based planning algorithms are used to obtain seeds, resulting in significant slowdowns. We propose DiffusionSeeder, a diffusion based approach that generates trajectories to seed motion optimization for rapid robot motion planning. DiffusionSeeder takes the initial depth image observation of the scene and generates high quality, multi-modal trajectories that are then fine-tuned with a few iterations of motion optimization. We integrate DiffusionSeeder to generate the seed trajectories for cuRobo, a GPU-accelerated motion optimization method, which results in 12x speed up on average, and 36x speed up for more complicated problems, while achieving 10% higher success rate in partially observed simulation environments. Our results show the effectiveness of using diverse solutions from a learned diffusion model. Physical experiments on a Franka robot demonstrate the sim2real transfer of DiffusionSeeder to the real robot, with an average success rate of 86% and planning time of 26ms, improving on cuRobo by 51% higher success rate while also being 2.5x faster.

Figures (7)

• Figure 1: We propose DiffusionSeeder, a diffusion based method that generates diverse, multi-modal trajectories to warm start motion optimization, greatly reducing the planning time on "hard problems". (A) The model takes depth images, the robot start configuration, and target pose. (B) DiffusionSeeder generates diverse multi-modal trajectories, where some trajectories could be in collision.A dynamic Euclidean Signed Distance Field (ESDF) using nvblox millane2023nvblox from the depth images is used by the motion optimizer cuRobo to move trajectories out of collision. (C) The final output from the motion optimizer is smooth, fast, and collision-free.
• Figure 2: DiffusionSeeder trajectory generation pipeline. DiffusionSeeder consists of a ViT observation encoder (blue) and a conditional diffusion model (coral). The robot start configuration and target pose are embedded through an MLP into a "problem encoding", which is concatenated with the environment embedding. The diffusion model conditions on the concatenated vector and a randomly-sampled noise tensor of size 32$\times$7 to generate initial trajectories for the optimizer (green). The optimizer takes in the ESDF for collision checking. The final trajectory after optimization is shown on the right.
• Figure 3: Planning time for cuRobo-100 and DiffusionSeeder-50 in partially and fully observed environment on the M$\pi$Net test set fishman2022mpinets
• Figure 4: Physical experiments scenes: (a) an easy scene with a box, (b) a hard scene where both cuRobo and DiffusionSeeder failed, (c) a medium scene. (c.1) shows cuRobo colliding while (c.2) shows DiffusionSeeder avoiding collision in the same scene.The start and end position are marked by the red stars. The two positions are to the left and right of the obstacles, requiring the robot to navigate around the obstacle for a safe execution.
• Figure 5: Example train (left) and test (right) problems on three scene types: cubby, tabletop, and dresser (from left to right in the left image). The robot is in the start configuration and the target end pose is marked in green.