Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems

Farnaz Adib Yaghmaie, Arunava Naha

TL;DR

The paper addresses convergence and stability of the one-step policy in Flow-$Q$-learning for offline Linear Quadratic Regulator (LQR) problems. It redefines the one-step policy learning as a policy-gradient problem by optimizing the average cost $J(\mu_K)$ with a behavioral-cloning regularizer, and proves that the resulting loss is $L$-smooth and gradient-dominant with respect to the gain $K$. The authors show that the learned policy gain $K$ converges linearly to the optimal $K^*$ and remains stabilizing under appropriate step sizes, with proofs supported by a linear-quadratic analysis. A linearized inverted pendulum experiment corroborates the theory, demonstrating that the method achieves near-optimal LQR performance while learning from offline data and performing competitively with scenario-based MPC. Overall, the work provides theoretical guarantees for Flow-$Q$-learning in linear control and lays groundwork for extensions to nonlinear dynamics and offline/federated settings.

Abstract

Flow $Q$-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure. Central to this innovation is the ``the one-step policy'' network, which is optimized through a $Q$-function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings. Our theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss. Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy. We verify our theoretical finding with simulation results on a linearized inverted pendulum.

Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems

TL;DR

The paper addresses convergence and stability of the one-step policy in Flow--learning for offline Linear Quadratic Regulator (LQR) problems. It redefines the one-step policy learning as a policy-gradient problem by optimizing the average cost with a behavioral-cloning regularizer, and proves that the resulting loss is -smooth and gradient-dominant with respect to the gain . The authors show that the learned policy gain converges linearly to the optimal and remains stabilizing under appropriate step sizes, with proofs supported by a linear-quadratic analysis. A linearized inverted pendulum experiment corroborates the theory, demonstrating that the method achieves near-optimal LQR performance while learning from offline data and performing competitively with scenario-based MPC. Overall, the work provides theoretical guarantees for Flow--learning in linear control and lays groundwork for extensions to nonlinear dynamics and offline/federated settings.

Abstract

Flow -learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure. Central to this innovation is the ``the one-step policy'' network, which is optimized through a -function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings. Our theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss. Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy. We verify our theoretical finding with simulation results on a linearized inverted pendulum.

Paper Structure

This paper contains 29 sections, 6 theorems, 49 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Notation and Flow $Q$-learning
  3. Notation and preliminaries
  4. Notation:
  5. L-smooth function:
  6. Gradient dominant function:
  7. MDP:
  8. Flow $Q$-learning
  9. The critic network $Q_{\varphi_c}$:
  10. The behavioral cloning (BC) policy $\mu_{\varphi_b}$:
  11. The one-step policy $\mu_{\varphi_o}$:
  12. Linear Quadratic Problem
  13. Steady state covariance of the state variable:
  14. Average expected cost:
  15. Flow-Policy Gradient Learning
  16. ...and 14 more sections

Key Result

Lemma 1

Consider the linear system in eq:lin:dynamics and assume that the controller gain $K \in \mathcal{K}$, where $\mathcal{K}=\{K \in \mathbb{R}^{m \times n} \mid \rho(A+BK) <1 \}$. Then, the covariance of the state variable $x_k$ under the policy $\pi = K x_k+z_k$ converges to a steady state value

Figures (3)

  • Figure 1: Convergence of the flow-matching loss (mean $\pm$ 95% CI).
  • Figure 2: Policy-gradient norm over epochs (mean $\pm$ 95% CI).
  • Figure 3: Total episodic costs: Proposed method vs. MPC (scenario-based vs. LQR. Shaded regions denote 95% CIs.

Theorems & Definitions (9)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • Theorem 1
  • Theorem 2
  • proof