Guided Policy Search using Sequential Convex Programming for Initialization of Trajectory Optimization Algorithms

TL;DR

The paper tackles the challenge of initializing nonlinear trajectory optimization under $\.dot{x}(t)=f(t,x(t),u(t))$ with state and input constraints and nonconvex constraints $s(t,x(t),u(t))\le 0$ by learning a neural policy that yields effective trajectory candidates ($u=\pi_\theta(x)$). It introduces a guided policy search framework that alternates a Penalized Trust Region (PTR)–based convex trajectory update with sampling around the updated trajectory via a time-varying LQR, followed by supervised learning to train $\pi_\theta$. Constraints are enforced through convex approximations and a policy-deviation penalty, and the method is validated on a 6-DoF minimum-fuel powered descent problem for a reusable rocket, demonstrating improved feasibility and convergence speed compared to traditional initialization. The approach enables fast, reliable initialization for high-dimensional trajectory optimization, with potential for real-time planning in safety-critical aerospace applications.

Abstract

Nonlinear trajectory optimization algorithms have been developed to handle optimal control problems with nonlinear dynamics and nonconvex constraints in trajectory planning. The performance and computational efficiency of many trajectory optimization methods are sensitive to the initial guess, i.e., the trajectory guess needed by the recursive trajectory optimization algorithm. Motivated by this observation, we tackle the initialization problem for trajectory optimization via policy optimization. To optimize a policy, we propose a guided policy search method that has two key components: i) Trajectory update; ii) Policy update. The trajectory update involves offline solutions of a large number of trajectory optimization problems from different initial states via Sequential Convex Programming (SCP). Here we take a single SCP step to generate the trajectory iterate for each problem. In conjunction with these iterates, we also generate additional trajectories around each iterate via a feedback control law. Then all these trajectories are used by a stochastic gradient descent algorithm to update the neural network policy, i.e., the policy update step. As a result, the trained policy makes it possible to generate trajectory candidates that are close to the optimality and feasibility and that provide excellent initial guesses for the trajectory optimization methods. We validate the proposed method via a real-world 6-degree-of-freedom powered descent guidance problem for a reusable rocket.

Figures (2)

• Figure 1: The block diagram of the proposed method. The method repeatedly solves two optimizations for the update of trajectory and policy. Between two optimizations, there is a step for generating samples that will be used for the policy update. Then, the trained policy generates a trajectory that will be an initial guess for trajectory optimization.
• Figure 2: The trajectory comparison between the proposed approach and the imitation learning method. The figures in the first column represent the trajectories starting from the initial states in the training set. The trajectories in the second column images and in the third column images start from the initial states in the validation set and the test set, respectively. The first and second rows show the same set of trajectories viewed from different angles. It is visible that the proposed approach generates less infeasible trajectories than the imitation learning method in terms of the boundary condition for the position.