Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimistic Reinforcement Learning with Quantile Objectives

Mohammad Alipour-Vaezi, Huaiyang Zhong, Kwok-Leung Tsui, Sajad Khodadadian

TL;DR

This work tackles risk-sensitive reinforcement learning by targeting the τ-quantile of finite-horizon returns in MDPs. It introduces UCB--QRL, a model-based optimistic algorithm that builds an ℓ1 confidence set over the transition kernel and uses a quantile-aware planner, aided by a continuation–mixture representation and a uniform quantile margin κ. A key theoretical contribution is a high-probability regret bound that explicitly depends on κ, the horizon, and problem size, demonstrating sublinear performance under the margin condition. The paper also discusses computational aspects, situates the approach among risk-sensitive and UCB-based RL literature, and outlines open directions such as function approximation and extensions to infinite-horizon settings. Overall, the results provide principled tail-risk control for sequential decision-making with finite-horizon dynamics and pave the way for robust, data-efficient quantile optimization in safety-critical domains.

Abstract

Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare and finance. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. In this paper, we develop UCB-QRL, an optimistic learning algorithm for the $τ$-quantile objective in finite-horizon Markov decision processes (MDPs). UCB-QRL is an iterative algorithm in which, at each iteration, we first estimate the underlying transition probability and then optimize the quantile value function over a confidence ball around this estimate. We show that UCB-QRL yields a high-probability regret bound $\mathcal O\left((2/κ)^{H+1}H\sqrt{SATH\log(2SATH/δ)}\right)$ in the episodic setting with $S$ states, $A$ actions, $T$ episodes, and $H$ horizons. Here, $κ>0$ is a problem-dependent constant that captures the sensitivity of the underlying MDP's quantile value.

Optimistic Reinforcement Learning with Quantile Objectives

TL;DR

This work tackles risk-sensitive reinforcement learning by targeting the τ-quantile of finite-horizon returns in MDPs. It introduces UCB--QRL, a model-based optimistic algorithm that builds an ℓ1 confidence set over the transition kernel and uses a quantile-aware planner, aided by a continuation–mixture representation and a uniform quantile margin κ. A key theoretical contribution is a high-probability regret bound that explicitly depends on κ, the horizon, and problem size, demonstrating sublinear performance under the margin condition. The paper also discusses computational aspects, situates the approach among risk-sensitive and UCB-based RL literature, and outlines open directions such as function approximation and extensions to infinite-horizon settings. Overall, the results provide principled tail-risk control for sequential decision-making with finite-horizon dynamics and pave the way for robust, data-efficient quantile optimization in safety-critical domains.

Abstract

Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare and finance. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. In this paper, we develop UCB-QRL, an optimistic learning algorithm for the -quantile objective in finite-horizon Markov decision processes (MDPs). UCB-QRL is an iterative algorithm in which, at each iteration, we first estimate the underlying transition probability and then optimize the quantile value function over a confidence ball around this estimate. We show that UCB-QRL yields a high-probability regret bound in the episodic setting with states, actions, episodes, and horizons. Here, is a problem-dependent constant that captures the sensitivity of the underlying MDP's quantile value.

Paper Structure

This paper contains 14 sections, 11 theorems, 115 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Risk-Sensitive MDPs
  5. Optimism and Upper Confidence Bounds (UCB)
  6. Preliminaries
  7. Finite Horizon UCB--QRL
  8. Discussion and Conclusion
  9. Computational Aspects
  10. Comparison with the Prior Work
  11. Assumption Discussion
  12. Appendix
  13. Proof of Theorem \ref{['thm:main']}
  14. Technical Lemmas

Key Result

Theorem 1

Let Assumption ass:margin hold with parameter $\kappa>0$. Then for UCB--QRL with confidence radii shown in Equation eq:confset, with probability at least $1-2\delta$,

Theorems & Definitions (23)

  • Definition 1: Continuation–mixture
  • Theorem 1: High-probability Quantile-Regret
  • proof
  • Lemma 1: High-probability optimism at episode $t$
  • proof : Proof of Lemma \ref{['lem:optimism-step0']}
  • Lemma 2: Weissman’s $\ell_1$ concentration
  • proof : Proof of Lemma \ref{['lem:weissman']}
  • Lemma 3: Bellman evaluation identity for the $\tau$–quantile
  • proof : Proof of Lemma \ref{['lem:bellman-eval']}
  • Lemma 4: Right identity for the left–continuous quantile
  • ...and 13 more