Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection

Chuhan Xie

TL;DR

The paper tackles risk-sensitive reinforcement learning in continuous time by representing dynamics with stochastic differential equations and risk functionals via optimized certainty equivalents (OCEs). It proves that OCE-based objectives yield Markovian optimal policies in an augmented state, enabling conventional RL methods; it then introduces CT-RS-q, a martingale-based on-policy Q-learning algorithm operating on augmented dynamics. The approach is demonstrated on a dynamic portfolio problem, where CT-RS-q achieves near-optimal mean-variance performance and outperforms baselines, validating the method's practicality. Overall, the work unifies continuous-time modeling with risk-sensitive objectives and provides a concrete algorithmic framework for finance and related domains.

Abstract

This paper studies the problem of risk-sensitive reinforcement learning (RSRL) in continuous time, where the environment is characterized by a controllable stochastic differential equation (SDE) and the objective is a potentially nonlinear functional of cumulative rewards. We prove that when the functional is an optimized certainty equivalent (OCE), the optimal policy is Markovian with respect to an augmented environment. We also propose \textit{CT-RS-q}, a risk-sensitive q-learning algorithm based on a novel martingale characterization approach. Finally, we run a simulation study on a dynamic portfolio selection problem and illustrate the effectiveness of our algorithm.

Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection

TL;DR

The paper tackles risk-sensitive reinforcement learning in continuous time by representing dynamics with stochastic differential equations and risk functionals via optimized certainty equivalents (OCEs). It proves that OCE-based objectives yield Markovian optimal policies in an augmented state, enabling conventional RL methods; it then introduces CT-RS-q, a martingale-based on-policy Q-learning algorithm operating on augmented dynamics. The approach is demonstrated on a dynamic portfolio problem, where CT-RS-q achieves near-optimal mean-variance performance and outperforms baselines, validating the method's practicality. Overall, the work unifies continuous-time modeling with risk-sensitive objectives and provides a concrete algorithmic framework for finance and related domains.

Abstract

This paper studies the problem of risk-sensitive reinforcement learning (RSRL) in continuous time, where the environment is characterized by a controllable stochastic differential equation (SDE) and the objective is a potentially nonlinear functional of cumulative rewards. We prove that when the functional is an optimized certainty equivalent (OCE), the optimal policy is Markovian with respect to an augmented environment. We also propose \textit{CT-RS-q}, a risk-sensitive q-learning algorithm based on a novel martingale characterization approach. Finally, we run a simulation study on a dynamic portfolio selection problem and illustrate the effectiveness of our algorithm.

Paper Structure

This paper contains 25 sections, 11 theorems, 59 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Resolving Nonlinearity of the Functional Objective
  4. OCE risk measures
  5. Markovian optimality
  6. Meta-algorithm for policy optimization
  7. Risk-Sensitive Q-Learning
  8. Martingale characterization
  9. Algorithm
  10. Application to Dynamic Portfolio Selection
  11. Theory of Risk-Sensitive Q-Learning
  12. Notation.
  13. Feynman-Kac formula and HJB equation
  14. Lowercase q-function
  15. Martingale characterization
  16. ...and 10 more sections

Key Result

Proposition 3.1

If the risk measure $U$ is an OCE with respect to the utility function $\varphi$, the optimal policy $\pi_0^* = \mathop{\rm argmax}_{\pi} J^{\pi}_0(t,x)$ of the SDE eq: sde-eq: J0 is Markovian with respect to the augmented state $(X_s, B_{0,s}, B_{1,s})$We have omitted the superscript $\pi$ on every

Figures (3)

  • Figure 1: Convergence of model parameters. The first three are parameters of the value function $J^\theta$, and the last five are parameters of the q-function $q^\psi$.
  • Figure 2: Curves of the cumulative return and the mean-variance objective for three policies.
  • Figure 3: Temporal difference of parameters around optimum (zeros in the plots above).

Theorems & Definitions (14)

  • Proposition 3.1
  • Theorem 4.1
  • Proposition A.1
  • Definition A.1
  • Proposition A.2
  • Proposition A.3
  • Proposition A.4
  • Theorem A.5
  • Theorem A.6
  • Theorem A.7
  • ...and 4 more