Shadow Program Inversion with Differentiable Planning: A Framework for Unified Robot Program Parameter and Trajectory Optimization

TL;DR

SPI-DP addresses the challenge of jointly optimizing robot program parameters and motion trajectories under motion-level constraints by unifying differentiable shadow programs with a differentiable planner. It introduces DGPMP2-ND, a differentiable motion planner for N-DoF serial manipulators, enabling gradient backpropagation through planning and integration into an end-to-end optimization of program inputs $\bm{x}$ and trajectories $\bm{\theta}$ with task-level objectives $\Phi$. The framework supports arbitrary parameterized robot programs via shadow graphs and optimizes objectives such as cycle time, path length, and success probability while enforcing collision-freeness and other constraints. Real-world experiments in household pick-and-place and industrial hole-search tasks demonstrate collision-free trajectories and substantial cycle-time reductions when jointly optimizing parameters and trajectories. This work offers a practical, differentiable approach to certifiable robot programming by optimization for complex, multi-skill tasks.

Abstract

This paper presents SPI-DP, a novel first-order optimizer capable of optimizing robot programs with respect to both high-level task objectives and motion-level constraints. To that end, we introduce DGPMP2-ND, a differentiable collision-free motion planner for serial N-DoF kinematics, and integrate it into an iterative, gradient-based optimization approach for generic, parameterized robot program representations. SPI-DP allows first-order optimization of planned trajectories and program parameters with respect to objectives such as cycle time or smoothness subject to e.g. collision constraints, while enabling humans to understand, modify or even certify the optimized programs. We provide a comprehensive evaluation on two practical household and industrial applications.

Figures (6)

• Figure 1: spi-dp enables the optimization of robot programs (left, red) by first-order iterative optimization over a differentiable surrogate (right, gray). A differentiable collision-free motion planner (dgpmp2-nd) ensures that the resulting motion trajectories are optimal with respect to task objectives and motion-level constraints.
• Figure 2: Left: spi-dp optimizes skill-based robot programs (left, red) with respect to nearly arbitrary task objectives $\Phi$. Program parameters, such as the Force parameter of a placing action, can be optimized jointly with low-level motion trajectories to respect task-level objectives and motion-level constraints. By performing gradient-based optimization over a differentiable surrogate ("shadow" $\bar{P}$, gray), the framework is applicable to near-arbitrary parameterized program representations, including most robot programming languages. Right: dgpmp2-nd plans collision-free motions for N-dof serial kinematics within spi-dp's optimization loop. Trajectories (black) are optimized with respect to collision (blue), goal (green), human demonstration (red) and other constraints.
• Figure 3: dgpmp2-nd permits motion planning with respect to joint-space and Cartesian constraints and objectives. It realizes trajectory optimization by iterative solving of a linear system, permitting the backpropagation of gradients through the planner.
• Figure 4: A differentiable shadow program for a search-based insertion task composed of two skills. By combining differentiable planners (blue) and trained neural networks (light green), program parameters $\bm{x}$ can be optimized with respect to task-level objectives $\Phi$ while respecting motion-level constraints such as collision-freeness. A forward pass (top to bottom) predicts the expected real-world trajectory given program parameters $\bm{x}$ and robot state $\theta$. The gradients of $\Phi$ are backpropagated and $\bm{x}$ is incrementally optimized.
• Figure 5: Left: Experiment \ref{['sec:cup-experiment-1']}: 3D rendering of the collision world, 4 exemplary optimization results (green) and 10 human demonstrations (red). Right: Experiment \ref{['sec:cup-experiment-2']}: Real-world execution of an optimized program.