Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach

Xin Guo, Zijiu Lyu

TL;DR

The paper advances policy transfer in continuous-time RL by establishing theoretical transfer results for entropy-regularized LQRs and general CT dynamics via Riccati stability and rough-path SDE stability, respectively. It introduces Iterative Policy Optimization (IPO) for LQRs, exploiting Gaussian optimality to achieve global linear and local super-linear convergence, enabling fast transfer to closely related LQRs. Additionally, the authors connect these dynamic-programming insights to the stability of score-based diffusion models through a Cole–Hopf transformation, providing concrete stability bounds. The work thus furnishes both rigorous guarantees and practical algorithms for TL in continuous-time RL, with implications for diffusion-model stability and broader stochastic control problems.

Abstract

This paper studies policy transfer, one of the well-known transfer learning techniques adopted in large language models, for two classes of continuous-time reinforcement learning problems. In the first class of continuous-time linear-quadratic systems with Shannon's entropy regularization (a.k.a. LQRs), we fully exploit the Gaussian structure of their optimal policy and the stability of their associated Riccati equations. In the second class where the system has possibly non-linear and bounded dynamics, the key technical component is the stability of diffusion SDEs which is established by invoking the rough path theory. Our work provides the first theoretical proof of policy transfer for continuous-time RL: an optimal policy learned for one RL problem can be used to initialize the search for a near-optimal policy in a closely related RL problem, while maintaining the convergence rate of the original algorithm. To illustrate the benefit of policy transfer for RL, we propose a novel policy learning algorithm for continuous-time LQRs, which achieves global linear convergence and local super-linear convergence. As a byproduct of our analysis, we derive the stability of a concrete class of continuous-time score-based diffusion models via their connection with LQRs.

Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach

TL;DR

The paper advances policy transfer in continuous-time RL by establishing theoretical transfer results for entropy-regularized LQRs and general CT dynamics via Riccati stability and rough-path SDE stability, respectively. It introduces Iterative Policy Optimization (IPO) for LQRs, exploiting Gaussian optimality to achieve global linear and local super-linear convergence, enabling fast transfer to closely related LQRs. Additionally, the authors connect these dynamic-programming insights to the stability of score-based diffusion models through a Cole–Hopf transformation, providing concrete stability bounds. The work thus furnishes both rigorous guarantees and practical algorithms for TL in continuous-time RL, with implications for diffusion-model stability and broader stochastic control problems.

Abstract

This paper studies policy transfer, one of the well-known transfer learning techniques adopted in large language models, for two classes of continuous-time reinforcement learning problems. In the first class of continuous-time linear-quadratic systems with Shannon's entropy regularization (a.k.a. LQRs), we fully exploit the Gaussian structure of their optimal policy and the stability of their associated Riccati equations. In the second class where the system has possibly non-linear and bounded dynamics, the key technical component is the stability of diffusion SDEs which is established by invoking the rough path theory. Our work provides the first theoretical proof of policy transfer for continuous-time RL: an optimal policy learned for one RL problem can be used to initialize the search for a near-optimal policy in a closely related RL problem, while maintaining the convergence rate of the original algorithm. To illustrate the benefit of policy transfer for RL, we propose a novel policy learning algorithm for continuous-time LQRs, which achieves global linear convergence and local super-linear convergence. As a byproduct of our analysis, we derive the stability of a concrete class of continuous-time score-based diffusion models via their connection with LQRs.
Paper Structure (29 sections, 19 theorems, 112 equations, 1 figure, 1 algorithm)

This paper contains 29 sections, 19 theorems, 112 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Reinforcement learning (RL).
  3. Transfer learning (TL).
  4. RL with TL.
  5. Our work.
  6. Related work.
  7. Organization of the paper.
  8. Notation.
  9. Transfer learning between RL problems
  10. Transfer learning between LQRs with entropy regularization
  11. Transfer learning between general continuous-time RL problems
  12. IPO and its local super-linear convergence for LQRs
  13. Iterative policy optimization (IPO) derivation.
  14. Convergence of IPO.
  15. Transfer learning with IPO.
  16. ...and 14 more sections

Key Result

Theorem 1

Given an LQR represented by model parameters $\theta$, for which there exists an optimal policy $\pi^*$ and an associated learning algorithm. Now, suppose there is another LQR represented by model parameters $\tilde{\theta}$. Then, there exists $\epsilon > 0$, such that with an appropriate initializ

Figures (1)

  • Figure 1: Basic mechanism of score-based diffusion models.

Theorems & Definitions (41)

  • Theorem 1: Transfer learning of LQRs
  • Lemma 2
  • Remark 1
  • Lemma 3: Stability of Riccati equation
  • Remark 2
  • Remark 3
  • Remark 4
  • Lemma 4
  • Lemma 5
  • Proposition 6: Well-posedness
  • ...and 31 more