Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions

TL;DR

This work develops a risk-sensitive RL framework built on convex scoring functions that unifies measures such as ES, variance, EVaR, and mean–risk utilities. To overcome time-inconsistency, it augments the state with an auxiliary variable and recasts the problem as a two-stage optimization: an inner time-consistent MDP solved via an Actor–Critic network and an outer optimization over the auxiliary variable $\\upsilon$, with a convergent alternating-minimization–inspired sampling scheme. The authors provide theoretical guarantees that do not require MDP continuity and establish convergence and approximation results for both the value function and the auxiliary-variable selection. They demonstrate the approach on a statistical arbitrage trading task, showing improved tail-risk control (lower $ES_{\\alpha}$) while maintaining competitive mean performance, and offer insights into policy structure and learning dynamics. Overall, the paper advances risk-sensitive RL by integrating convex scoring risk measures with a tractable two-stage optimization and provable convergence, with practical impact for financial decision-making under tail risk constraints.

Abstract

We propose a reinforcement learning (RL) framework under a broad class of risk objectives, characterized by convex scoring functions. This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility. To resolve the time-inconsistency issue, we consider an augmented state space and an auxiliary variable and recast the problem as a two-state optimization problem. We propose a customized Actor-Critic algorithm and establish some theoretical approximation guarantees. A key theoretical contribution is that our results do not require the Markov decision process to be continuous. Additionally, we propose an auxiliary variable sampling method inspired by the alternating minimization algorithm, which is convergent under certain conditions. We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.

Figures (13)

• Figure 1: Proposed Learning Procedure
• Figure 2: Approximation of the value function
• Figure 3: RL-mean
• Figure 4: RL-OneStep$\mathrm{ES}_{0.8}$
• Figure 5: RL-$\mathrm{ES}_{0.8}$

Theorems & Definitions (35)

• Definition 1: Convex scoring function
• Lemma 1
• Example 1
• Example 2
• Remark 1
• Proposition 1
• Remark 2
• Remark 3
• Theorem 1: Bellman Equation
• Remark 4