Learning-based Initialization of Trajectory Optimization for Path-following Problems of Redundant Manipulators

TL;DR

This work tackles the challenge of initializing trajectory optimization for path-following with redundant manipulators, where initial guesses heavily affect convergence and final solution quality. It proposes RL-ITG, an example-guided reinforcement learning approach that uses a path-conditioned MDP and a null-space aware imitation reward to generate high-quality initial joint trajectories offline. Through large-scale simulation and real-robot experiments on a seven-DoF Fetch arm, RL-ITG demonstrates improved TO optimality, faster convergence, and broader applicability across problem variations, compared to several baselines. The approach promises practical impact by amortizing the heavy online planning load and enabling reliable, efficient long-horizon planning in static environments, with plans to extend to dynamic scenes via receding-horizon control.

Abstract

Trajectory optimization (TO) is an efficient tool to generate a redundant manipulator's joint trajectory following a 6-dimensional Cartesian path. The optimization performance largely depends on the quality of initial trajectories. However, the selection of a high-quality initial trajectory is non-trivial and requires a considerable time budget due to the extremely large space of the solution trajectories and the lack of prior knowledge about task constraints in configuration space. To alleviate the issue, we present a learning-based initial trajectory generation method that generates high-quality initial trajectories in a short time budget by adopting example-guided reinforcement learning. In addition, we suggest a null-space projected imitation reward to consider null-space constraints by efficiently learning kinematically feasible motion captured in expert demonstrations. Our statistical evaluation in simulation shows the improved optimality, efficiency, and applicability of TO when we plug in our method's output, compared with three other baselines. We also show the performance improvement and feasibility via real-world experiments with a seven-degree-of-freedom manipulator.

Figures (8)

• Figure 1: Given an exemplar path-following problem 'Hello', first-row figures show initial trajectories of two baselines, Linearschulman2013findingzucker2013chomp and Greedyluo2012interactivekang2020torm, and ours, RL-ITG. Putting these into a trajectory optimizer (TO), second-row figures show optimization results. Ours quickly generates a high-quality initial trajectory rapidly converging to a better local optimum while the others get stuck in a boundary of a feasible solution. We return the solution of TO when the error is below 0.001 or the time limit of 50s is exceeded.
• Figure 2: Illustration of how RL-ITG produces an initial trajectory $\xi_{init}$. Starting from a start configuration $q_{0}$, our policy takes as input task-space information (i.e., local target poses, robot states, and a 3-dimensional occupancy map) and sequentially expands the trajectory in configuration space, taking into account objectives and constraints related to the path-following problem.
• Figure 3: Learning curves of four reward function compositions. We measure the success rate on 20 evaluation sets randomly generated with the procedure in Sec. \ref{['sec:3-B']} at every $10^4$ steps and consider one experiment successful when distances between the end effector pose and the target pose are within 5 positionally and 3° rotationally without any collision at all time steps.
• Figure 4: Visualization of five specific and two exemplar random target paths (red lines) used in evaluations. Orange arrows indicate the progress direction of the paths. Blue lines are the end-effector paths of the initial joint trajectories generated by our method, RL-ITG. In 'Square', 'S', and 'Random w/ obs', the original color of the robot represents the start configuration, and the yellow trail (i.e., some intermediate configurations) shows that the trajectories avoid collisions with external objects and the robot body via optimizing the null-space motion while precisely following the paths.
• Figure 5: Comparison of average pose error convergence of four initialization methods over eight combination experiments of four benchmark sets (i.e., Hello, Rotation, Zigzag, and Square & S) and two optimization methods (i.e., TORM kang2020torm and TrajOpt schulman2013finding). For each sub-figure, the y-axis is the pose error in the log scale, the x-axis is the elapsed time (s), and the shaded area represents one standard deviation over the mean.