Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise

Gabriel Diaz, Lucky Li, Wenhao Zhang

TL;DR

Problem: entropy-regularized, infinite-horizon LQ control with multiplicative noise. Approach: adapt regularized policy gradient to stochastic LQ with multiplicative noise and develop a model-free SB-RPG using zero-order gradient estimates, both with provable global convergence under gradient domination and almost-smoothness. Contributions: global convergence results for RPG (model-based) and SB-RPG (model-free), explicit gradient forms, contraction analysis, and numerical validation demonstrating practicality in unknown-parameter settings. Significance: extends RL control methods to robust, exploration-aware policies under multiplicative disturbances and unknown system parameters, with rigorous convergence guarantees and practical performance.

Abstract

Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entropy-regularized linear-quadratic (LQ) control problems with multiplicative noise over an infinite time horizon. First, we adapt the regularized policy gradient (RPG) algorithm to stochastic optimal control settings, proving that despite the non-convexity of the problem, RPG converges globally under conditions of gradient domination and almost-smoothness. Second, based on zero-order optimization approach, we introduce a novel model free RL algorithm: Sample-based regularized policy gradient (SB-RPG). SB-RPG operates without knowledge of system parameters yet still retains strong theoretical guarantees of global convergence. Our model leverages entropy regularization to address the exploration versus exploitation trade-off inherent in RL. Numerical simulations validate the theoretical results and demonstrate the efficiency of SB-RPG in unknown-parameters environments.

Global Convergence of Policy Gradient for Entropy Regularized Linear-Quadratic Control with Multiplicative Noise

TL;DR

Problem: entropy-regularized, infinite-horizon LQ control with multiplicative noise. Approach: adapt regularized policy gradient to stochastic LQ with multiplicative noise and develop a model-free SB-RPG using zero-order gradient estimates, both with provable global convergence under gradient domination and almost-smoothness. Contributions: global convergence results for RPG (model-based) and SB-RPG (model-free), explicit gradient forms, contraction analysis, and numerical validation demonstrating practicality in unknown-parameter settings. Significance: extends RL control methods to robust, exploration-aware policies under multiplicative disturbances and unknown system parameters, with rigorous convergence guarantees and practical performance.

Abstract

Reinforcement Learning (RL) has emerged as a powerful framework for sequential decision-making in dynamic environments, particularly when system parameters are unknown. This paper investigates RL-based control for entropy-regularized linear-quadratic (LQ) control problems with multiplicative noise over an infinite time horizon. First, we adapt the regularized policy gradient (RPG) algorithm to stochastic optimal control settings, proving that despite the non-convexity of the problem, RPG converges globally under conditions of gradient domination and almost-smoothness. Second, based on zero-order optimization approach, we introduce a novel model free RL algorithm: Sample-based regularized policy gradient (SB-RPG). SB-RPG operates without knowledge of system parameters yet still retains strong theoretical guarantees of global convergence. Our model leverages entropy regularization to address the exploration versus exploitation trade-off inherent in RL. Numerical simulations validate the theoretical results and demonstrate the efficiency of SB-RPG in unknown-parameters environments.

Paper Structure

This paper contains 33 sections, 25 theorems, 184 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Formulation
  3. Global Convergence of Regularized Policy Gradient
  4. Global Convergence of Sample Based Regularized Policy Gradient
  5. Numerical Experiments
  6. Proofs in Section \ref{['section: formulation']}
  7. Proof of Lemma \ref{['lemma: solve regularzied QP']}
  8. Proof of Lemma \ref{['lemma: optimization']}
  9. Proof of Lemma \ref{['lemma: explicit form']}
  10. Proofs in Section \ref{['section: model based']}
  11. Proof of Lemma \ref{['lemma: gradient domination']}
  12. Proof of Lemma \ref{['lemma: norm bounds']}
  13. Proof of Lemma \ref{['lemma: almost smooth']}
  14. Proof of Lemma \ref{['lemma: bound of sigma']}
  15. Proof of Lemma \ref{['lemma: contraction of RPG']}
  16. ...and 18 more sections

Key Result

Theorem 2.1

The optimal value function $J^*:\mathcal{X} \rightarrow \mathbb{R}$ in can be expressed as $J^*(x) = Px^2 + q$ with $P$ satisfying the following Algebraic Riccati Equation (ARE) where for any $x \in \mathcal{X}$, the corresponding optimal policy for system eq:state and objective functional eqn: obj initial is :

Figures (2)

  • Figure 1: Expected cost of RPG
  • Figure 2: Squared error between optimal policy and SB-RPG

Theorems & Definitions (49)

  • Theorem 2.1: Optimal value function and optimal control
  • Lemma 2.1
  • proof
  • Lemma 2.2: Optimization formulation
  • Lemma 2.3: Explicit form of $\nabla _Kf(K,\Sigma)$ and $\nabla _\Sigma f(K,\Sigma)$
  • Lemma 3.1: Gradient Domination of $f(K,\Sigma)$
  • Lemma 3.2: Gradient Norm Bounds
  • Lemma 3.3: "Almost" smoothness of $f(K,\Sigma)$
  • Lemma 3.4: Boundedness of update $\Sigma$
  • Lemma 3.5: Contraction of RPG
  • ...and 39 more