Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous-time q-learning for Markov regime switching system under Tsallis entropy

Minghui Zhang, Xun Li, Xin Zhang

TL;DR

This work develops a continuous-time reinforcement learning framework for Markov regime-switching systems with Tsallis entropy regularization, addressing the challenge that the regularized optimal policy need not be Gibbs. The authors establish a martingale characterization of the q-function under Tsallis entropy and design two model-free q-learning algorithms that differ in whether the normalizing function $\psi$ is available. They apply the methods to a continuous-time exploratory Mean-Variance portfolio problem in a two-regime market, demonstrating convergence of learned parameters to ground-truth values for both $p=1$ and $p=2$ and illustrating the practical viability of the approach. The paper thus provides a general, regulator-tolerant framework for continuous-time regime-switching RL with flexible entropy regularization, enabling robust exploration and policy learning beyond Gaussian/shannon-based formulations.

Abstract

This paper studies the continuous-time q-learning (the continuous time counterpart of Q-learing) for Markov switching system under Tsallis entropy regularization. We address the difficulty in traditional RL algorithms where the Tsallis entropy regularization leads to an optimal policy distribution not necessarily a Gibbs measure, which often complicates algorithm design. Furthermore, to address the limited universality of current continuous time regime-switching RL algorithms (often restricted to the EMV framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time RL method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. Based on this, we design two q-learning algorithms, distinguished by whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory Mean-Variance (EMV) portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.

Continuous-time q-learning for Markov regime switching system under Tsallis entropy

TL;DR

This work develops a continuous-time reinforcement learning framework for Markov regime-switching systems with Tsallis entropy regularization, addressing the challenge that the regularized optimal policy need not be Gibbs. The authors establish a martingale characterization of the q-function under Tsallis entropy and design two model-free q-learning algorithms that differ in whether the normalizing function is available. They apply the methods to a continuous-time exploratory Mean-Variance portfolio problem in a two-regime market, demonstrating convergence of learned parameters to ground-truth values for both and and illustrating the practical viability of the approach. The paper thus provides a general, regulator-tolerant framework for continuous-time regime-switching RL with flexible entropy regularization, enabling robust exploration and policy learning beyond Gaussian/shannon-based formulations.

Abstract

This paper studies the continuous-time q-learning (the continuous time counterpart of Q-learing) for Markov switching system under Tsallis entropy regularization. We address the difficulty in traditional RL algorithms where the Tsallis entropy regularization leads to an optimal policy distribution not necessarily a Gibbs measure, which often complicates algorithm design. Furthermore, to address the limited universality of current continuous time regime-switching RL algorithms (often restricted to the EMV framework), this study focuses on continuous-time q-learning for Markov regime-switching systems based on Tsallis entropy, aiming for a more universally applicable continuous-time RL method. We establish the martingale characterization of the q-function under Tsallis entropy for continuous-time Markov regime-switching systems. Based on this, we design two q-learning algorithms, distinguished by whether the Lagrange multiplier can be explicitly derived. We apply these algorithms to the continuous-time exploratory Mean-Variance (EMV) portfolio optimization problem in a regime-switching market. Numerical experiments demonstrate the satisfactory performance of our q-learning algorithms.
Paper Structure (13 sections, 9 theorems, 131 equations, 2 figures, 3 algorithms)

This paper contains 13 sections, 9 theorems, 131 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Problem formulation
  4. Exploratory control problem
  5. The Exploratory HJB equation
  6. Continuous-time q-function for Markov regime switching system and Martingale Characterization under Tsallis Entropy
  7. q-Learning Algorithms under Tsallis Entropy
  8. q-Learning algorithm when the normalizing function is available
  9. q-Learning algorithm when the normalizing function is unavailable
  10. Applications and Numerical Examples
  11. The case of p equals 1
  12. The case of p greater than 1
  13. Conclusion

Key Result

Lemma 2.3

Let Assumptions assumption hold and $\boldsymbol{\pi}$ be a given admissible policy. Then the $S D E$eq:state-Exp admits a unique strong solution. Moreover, for any $n \geq 2$, the solution satisfies the growth condition $\mathbb{E} ^{\mathbb{P}^W}\left[\max _{t \leq s \leq T}|\tilde{X}_s^\pi|^n \mi

Figures (2)

  • Figure 1: Convergence of Algorithm \ref{['Alg:Tsallis-q-Learning']} using a market simulator. The panels show the convergence of parameter iterations for ($\rho_1,\rho_2,\sigma_1,\sigma_2$).
  • Figure 2: Convergence of Algorithm \ref{['Alg:Tsallis-q-Learning-normalizing-unavailable-F']} using a market simulator. The panels show the convergence of parameter iterations for ($\rho_1,\rho_2,\sigma_1,\sigma_2$).

Theorems & Definitions (20)

  • Remark 2.2
  • Definition 2.1
  • Lemma 2.3
  • Theorem 2.4: Policy Improvement Iteration
  • Proposition 3.1
  • Definition 3.1: q-function
  • Corollary 3.2
  • Remark 3.3
  • Proposition 3.4
  • Theorem 3.5
  • ...and 10 more