Accelerated Optimization Landscape of Linear-Quadratic Regulator
Lechen Feng, Yuan-Hua Ni
TL;DR
The paper addresses accelerating first-order optimization for linear-quadratic regulators (LQR), handling both state-feedback (SLQR) and output-feedback (OLQR) forms. It proves a Lipschitz Hessian property for the LQR cost and develops an accelerated continuous-time framework with a restarting hybrid dynamics that achieves the Nesterov-optimal rate $1-\frac{1}{\sqrt{\kappa}}$ for SLQR, with a discretized scheme that preserves this rate. For OLQR, it introduces a Hessian-free framework combining semiconvex optimization and negative curvature exploitation, yielding an $\mathcal{O}(\epsilon^{-7/4}\log(1/\epsilon))$ time to an $\epsilon$-stationary point with a second-order guarantee. The approach leverages sublevel-set smoothness, restarting rules, and a penalty-based augmentation to manage nonconvexity, delivering both theoretical convergence rates and practical numerical demonstrations. Together, these results offer a unified, efficient first-order optimization toolbox for LQR policy design with provable acceleration and robustness to nonconvexity.
Abstract
Linear-quadratic regulator (LQR) is a landmark problem in the field of optimal control, which is the concern of this paper. Generally, LQR is classified into state-feedback LQR (SLQR) and output-feedback LQR (OLQR) based on whether the full state is obtained. It has been suggested in existing literature that both SLQR and OLQR could be viewed as \textit{constrained nonconvex matrix optimization} problems in which the only variable to be optimized is the feedback gain matrix. In this paper, we introduce a first-order accelerated optimization framework of handling the LQR problem, and give its convergence analysis for the cases of SLQR and OLQR, respectively. Specifically, a Lipschiz Hessian property of LQR performance criterion is presented, which turns out to be a crucial property for the application of modern optimization techniques. For the SLQR problem, a continuous-time hybrid dynamic system is introduced, whose solution trajectory is shown to converge exponentially to the optimal feedback gain with Nesterov-optimal order $1-\frac{1}{\sqrtκ}$ ($κ$ the condition number). Then, the symplectic Euler scheme is utilized to discretize the hybrid dynamic system, and a Nesterov-type method with a restarting rule is proposed that preserves the continuous-time convergence rate, i.e., the discretized algorithm admits the Nesterov-optimal convergence order. For the OLQR problem, a Hessian-free accelerated framework is proposed, which is a two-procedure method consisting of semiconvex function optimization and negative curvature exploitation. In a time $\mathcal{O}(ε^{-7/4}\log(1/ε))$, the method can find an $ε$-stationary point of the performance criterion; this entails that the method improves upon the $\mathcal{O}(ε^{-2})$ complexity of vanilla gradient descent. Moreover, our method provides the second-order guarantee of stationary point.