Revisiting Strong Duality, Hidden Convexity, and Gradient Dominance in the Linear Quadratic Regulator

TL;DR

The paper reexamines the Linear Quadratic Regulator through a primal–dual lens, revealing strong duality for the nonconvex policy problem under stabilizability and detectability, and characterizing when gradient dominance holds via Extended Convex Lifting (ECL) and Cauchy directions. It then shows that static linear policies are globally optimal among all stabilizing policies by leveraging Gramian/covariance representations and SDP-based inner/outer relaxations, with the optimal gain recoverable from the Riccati solution. Collectively, these results connect classical Riccati theory, SDP duality, and modern nonconvex optimization insights to provide a deeper, more unified understanding of LQR landscapes. The findings offer theoretical grounding for model-free and first-order algorithms in LQR and hint at broader applicability to other control problems where convex reformulations and lifting strategies can reveal hidden structure and convergence guarantees.

Abstract

The Linear Quadratic Regulator (LQR) is a cornerstone of optimal control theory, widely studied in both model-based and model-free approaches. Despite its well-established nature, certain foundational aspects remain subtle. In this paper, we revisit three key properties of policy optimization in LQR: (i) strong duality in the nonconvex policy optimization formulation, (ii) the gradient dominance property, examining when it holds and when it fails, and (iii) the global optimality of linear static policies. Using primal-dual analysis and convex reformulation, we refine and clarify existing results by leveraging Riccati equations/inequalities, semidefinite programming (SDP) duality, and a recent framework of Extended Convex Lifting (\texttt{ECL}). Our analysis confirms that LQR 1) behaves almost like a convex problem (e.g., strong duality) under the standard assumptions of stabilizability and detectability and 2) exhibits strong convexity-like properties (e.g., gradient dominance) under slightly stronger conditions. In particular, we establish a broader characterization under which gradient dominance holds using \texttt{ECL} and the notion of Cauchy directions. By clarifying and refining these theoretical insights, we hope this work contributes to a deeper understanding of LQR and may inspire further developments beyond LQR.

Figures (6)

• Figure 1: Nonconvex landscapes in LQR: (a) a slice of the set of static feedback gains $K$ such that $A+BK$ is stable, where $A=0$ and $B=I_2$; (b) a "well-behaved" nonconvex and smooth landscape of the LQR cost from zheng2023benign; (c) Non-unique globally optimal LQR feedback gains in \ref{['example:W-singular']}; sublevel sets of this instance are unbounded. The red point $K^\star$ denotes the optimal feedback gain from solving the ARE.
• Figure 2: The slices of $J_\mathtt{LQR}([k_1,k_2])$ with $k_2=-b-k_1$ in \ref{['example:W-singular']} for $b=1$, $10$, and $100$. We set the range of $k_1$ as $k_1\in\left[-5-{b}/{2},5-{b}/{2}\right]$ for each $b$. As $b$ increases, the local shape becomes flatter.
• Figure 3: A "well-behaved" nonconvex LQR landscape for \ref{['example:PL-W>=0']} with $a=0.1$ that satisfies \ref{['assumption:compactness']}. By \ref{['theorem:J_LQR-gradient_dominance-general']}, the optimal feedback gain is unique, and gradient dominance is satisfied over $\mathcal{K}_\nu$.
• Figure 4: Illustration of Cauchy directions. (a) Case 1: $h_1(x,y)=2x^2+y^2$. The blue and green arrows denote $\nabla h_1(x,y)=[4x,2y]^{{\mathsf T}}$ and the Cauchy direction $g_\mathrm{c}$ in \ref{['eq:Cauchy-direction-differentiable-h']} at $(1,1)$, respectively; (b) Case 2: $h_2(x)=x^2-x-3$. The red arrow is $[\nabla h(2),-1]x-2y+1=0$ with $\nabla h_2(2)=3$, and blue arrow is $[g_\mathrm{c},-1]x-2y+1=0$.
• Figure 5: Illustration of the inclusion $\mathcal{V}_\mathrm{static} \subset \mathcal{V} \subseteq \mathcal{V}_\mathrm{sdp}$ in \ref{['example:V-inclusion']}. The shaded area denotes the convex set $\mathcal{V}_\mathrm{sdp}$, and the grey area sketches $\mathcal{V}$, which might be nonconvex. The solid black line denotes the boundary of $\mathcal{V}_\mathrm{sdp}$, which is the same as $\mathcal{V}_\mathrm{static}$. From \ref{['theorem:linear-optimality']}, the optimal $Z^\star\in\mathcal{V}_\mathrm{sdp}$ to \ref{['eq:SDP-primal-w/x_0']} is attained on $\mathcal{V}_\mathrm{static}$.

Theorems & Definitions (56)

• Lemma 1
• Remark 1: Positive (semi)definiteness of $W$
• Theorem 1: Strong duality
• Theorem 2: Primal and dual optimal solutions
• Remark 2: Uniqueness of the optimal feedback gain
• Example 1: Non-unique optimal feedback gains
• Lemma 2
• Theorem 3
• Lemma 3
• Lemma 4