Fast Policy Learning for Linear Quadratic Control with Entropy Regularization

TL;DR

This work develops two policy-learning methods, RPG and IPO, for entropy-regularized linear-quadratic control over infinite horizons, proving global linear convergence for both and local super-linear convergence for IPO near the optimum. It shows that the optimal policy is Gaussian with mean linear in the state and a fixed covariance, and derives explicit expressions for both the policy gradient and the state-correlation dynamics. The analysis introduces gradient-dominance and an 'almost' smoothness property to support convergence proofs, and demonstrates transfer-learning advantages when initializing from a closely related environment. Numerically, increasing the entropy regularization parameter \tau and dynamically updating the covariance \Sigma significantly accelerate convergence, with IPO delivering particularly fast, even super-linear, progress in practice.

Abstract

This paper proposes and analyzes two new policy learning methods: regularized policy gradient (RPG) and iterative policy optimization (IPO), for a class of discounted linear-quadratic control (LQC) problems over an infinite time horizon with entropy regularization. Assuming access to the exact policy evaluation, both proposed approaches are proven to converge linearly in finding optimal policies of the regularized LQC. Moreover, the IPO method can achieve a super-linear convergence rate once it enters a local region around the optimal policy. Finally, when the optimal policy for an RL problem with a known environment is appropriately transferred as the initial policy to an RL problem with an unknown environment, the IPO method is shown to enable a super-linear convergence rate if the two environments are sufficiently close. Performances of these proposed algorithms are supported by numerical examples.

Figures (3)

• Figure 1: Performances of (\ref{['fig:reguPG']}) \ref{['alg: regularized update']}; (\ref{['fig:unperturbed']}) \ref{['alg: PI']}; (\ref{['fig:perturbed']}) transfer learning using \ref{['alg: PI']} with $(\overline{K}^{(0)},\overline{\Sigma}^{(0)}) = (K^*,\Sigma^*)$ and state transitions $(\overline{A}, \overline{B})$. $n=400$, $k=200.$ The regularization parameter $\tau$ is chosen to be $\sigma_{\min}(R)$.
• Figure 2: Regularized Policy Gradient \ref{['alg: regularized update']} with different regularization parameters $\tau$. Left: $n=200, k =10;$ Right: $n=200, k =50.$
• Figure 3: Comparison between Iterative Policy Optimization \ref{['alg: PI']} and Gauss-Newton update on $K$ with constant covariance matrix \ref{['eq: Gauss-Newton']}. $n=200, k=50$. Left: $\tau = 0.01;$ Right: $\tau = 0.5.$

Theorems & Definitions (42)

• Theorem 2.1: Optimal value functions and optimal policy
• Lemma 2.2
• Proof 1
• Lemma 3.1: Explicit form for the policy gradient
• Lemma 3.2: Gradient dominance of $C(K,\Sigma)$
• Lemma 3.3: "Almost" smoothness of $C(K, \Sigma)$
• Remark 4.1: Comparison to natural policy gradient
• Theorem 4.2: Global convergence of \ref{['alg: regularized update']}
• Remark 4.3
• Remark 4.4