Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

Lucky Li

TL;DR

This work develops a reinforcement learning approach for a discrete-time linear-quadratic control problem with unknown dynamics and an entropy-regularized objective, showing that the optimal feedback policy is Gaussian and deriving explicit backward recursions akin to Riccati equations. It then applies these results to a mean-variance asset-liability management setting, establishing policy improvement and convergence guarantees for the learning algorithm and demonstrating practical performance through numerical experiments with monthly and daily rebalancing. The contributions include a complete RL scheme that estimates unknown dynamics and a dynamic Lagrange multiplier, along with rigorous convergence results and illustrative MV problem applications. The work lays groundwork for online, data-driven control in finance and signals a path toward extending the framework to nonlinear systems via policy-iteration techniques.

Abstract

We study the discrete-time linear-quadratic (LQ) control model using reinforcement learning (RL). Using entropy to measure the cost of exploration, we prove that the optimal feedback policy for the problem must be Gaussian type. Then, we apply the results of the discrete-time LQ model to solve the discrete-time mean-variance asset-liability management problem and prove our RL algorithm's policy improvement and convergence. Finally, a numerical example sheds light on the theoretical results established using simulations.

Reinforcement Learning for a Discrete-Time Linear-Quadratic Control Problem with an Application

TL;DR

This work develops a reinforcement learning approach for a discrete-time linear-quadratic control problem with unknown dynamics and an entropy-regularized objective, showing that the optimal feedback policy is Gaussian and deriving explicit backward recursions akin to Riccati equations. It then applies these results to a mean-variance asset-liability management setting, establishing policy improvement and convergence guarantees for the learning algorithm and demonstrating practical performance through numerical experiments with monthly and daily rebalancing. The contributions include a complete RL scheme that estimates unknown dynamics and a dynamic Lagrange multiplier, along with rigorous convergence results and illustrative MV problem applications. The work lays groundwork for online, data-driven control in finance and signals a path toward extending the framework to nonlinear systems via policy-iteration techniques.

Abstract

We study the discrete-time linear-quadratic (LQ) control model using reinforcement learning (RL). Using entropy to measure the cost of exploration, we prove that the optimal feedback policy for the problem must be Gaussian type. Then, we apply the results of the discrete-time LQ model to solve the discrete-time mean-variance asset-liability management problem and prove our RL algorithm's policy improvement and convergence. Finally, a numerical example sheds light on the theoretical results established using simulations.

Paper Structure

This paper contains 18 sections, 6 theorems, 116 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Formulation
  3. Mean--Variance Model of Asset-liability Management
  4. Reinforcement Learning Algorithm
  5. Policy improvement and policy convergence
  6. Algorithm Design
  7. Initialization
  8. Policy Evaluation and Policy Improvement
  9. Lagrange Multiplier
  10. Numerical Result
  11. Monthly Rebalancing
  12. Daily Rebalancing
  13. Conclusion
  14. Lemma
  15. Proof of Theorem 1
  16. ...and 3 more sections

Key Result

Theorem 1

At period $t$, the optimal value function can be presented by and the corresponding optimal feedback control $u$ follows Gaussian distribution, with its density function given by where for $t=0,1,\cdots T-1$ and $P_T = Q_T$.

Figures (2)

  • Figure 1: Chang of Terminal Surplus, Lagrange Multiplier and Bellman Error in a monthly rebalancing (left: 1 year, right: 5years)
  • Figure 2: Chang of Terminal Surplus, Lagrange Multiplier and Bellman Error in a daily rebalancing (left: half year, right: 1 year)

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Proposition 1
  • Lemma 1
  • Theorem 3
  • Lemma 2