Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning

Dake Zhang, Boxiang Lyu, Shuang Qiu, Mladen Kolar, Tong Zhang

TL;DR

This work develops the first provably efficient offline reinforcement learning methods for risk-sensitive objectives under the entropic risk measure $V_\beta = \frac{1}{\beta}\log\{\mathbb{E}[e^{\beta R}]\}$ in linear MDPs. It introduces two pessimistic algorithms: (i) Risk-Sensitive Pessimistic Value Iteration (RSPVI), which uses a pessimistic bonus guided by the entropic structure, and (ii) Variance-Aware RSPVI (VA-RSPVI), which augments estimation with variance information via an auxiliary dataset. Theoretical guarantees show suboptimality bounds that depend on the risk-sensitivity factor $\frac{e^{|\beta|H}-1}{|\beta|}$ and feature dimension $d$, with improved $\sqrt{d}$-type rates under data coverage through reference-advantage decomposition and variance-aware techniques. As $\beta \to 0$, the results recover the risk-neutral offline RL rates, aligning with existing linear MDP bounds. The analysis leverages the exponential Bellman framework, a shifting/scaling transformation, and covering-number bounds to handle the risk-sensitive setting, making these results potentially generalizable to broader offline risk-sensitive RL problems.

Abstract

We study risk-sensitive reinforcement learning (RL), a crucial field due to its ability to enhance decision-making in scenarios where it is essential to manage uncertainty and minimize potential adverse outcomes. Particularly, our work focuses on applying the entropic risk measure to RL problems. While existing literature primarily investigates the online setting, there remains a large gap in understanding how to efficiently derive a near-optimal policy based on this risk measure using only a pre-collected dataset. We center on the linear Markov Decision Process (MDP) setting, a well-regarded theoretical framework that has yet to be examined from a risk-sensitive standpoint. In response, we introduce two provably sample-efficient algorithms. We begin by presenting a risk-sensitive pessimistic value iteration algorithm, offering a tight analysis by leveraging the structure of the risk-sensitive performance measure. To further improve the obtained bounds, we propose another pessimistic algorithm that utilizes variance information and reference-advantage decomposition, effectively improving both the dependence on the space dimension $d$ and the risk-sensitivity factor. To the best of our knowledge, we obtain the first provably efficient risk-sensitive offline RL algorithms.

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning

TL;DR

This work develops the first provably efficient offline reinforcement learning methods for risk-sensitive objectives under the entropic risk measure in linear MDPs. It introduces two pessimistic algorithms: (i) Risk-Sensitive Pessimistic Value Iteration (RSPVI), which uses a pessimistic bonus guided by the entropic structure, and (ii) Variance-Aware RSPVI (VA-RSPVI), which augments estimation with variance information via an auxiliary dataset. Theoretical guarantees show suboptimality bounds that depend on the risk-sensitivity factor and feature dimension , with improved -type rates under data coverage through reference-advantage decomposition and variance-aware techniques. As , the results recover the risk-neutral offline RL rates, aligning with existing linear MDP bounds. The analysis leverages the exponential Bellman framework, a shifting/scaling transformation, and covering-number bounds to handle the risk-sensitive setting, making these results potentially generalizable to broader offline risk-sensitive RL problems.

Abstract

We study risk-sensitive reinforcement learning (RL), a crucial field due to its ability to enhance decision-making in scenarios where it is essential to manage uncertainty and minimize potential adverse outcomes. Particularly, our work focuses on applying the entropic risk measure to RL problems. While existing literature primarily investigates the online setting, there remains a large gap in understanding how to efficiently derive a near-optimal policy based on this risk measure using only a pre-collected dataset. We center on the linear Markov Decision Process (MDP) setting, a well-regarded theoretical framework that has yet to be examined from a risk-sensitive standpoint. In response, we introduce two provably sample-efficient algorithms. We begin by presenting a risk-sensitive pessimistic value iteration algorithm, offering a tight analysis by leveraging the structure of the risk-sensitive performance measure. To further improve the obtained bounds, we propose another pessimistic algorithm that utilizes variance information and reference-advantage decomposition, effectively improving both the dependence on the space dimension and the risk-sensitivity factor. To the best of our knowledge, we obtain the first provably efficient risk-sensitive offline RL algorithms.
Paper Structure (27 sections, 15 theorems, 143 equations, 1 figure, 2 algorithms)

This paper contains 27 sections, 15 theorems, 143 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Suboptimality Analysis of Risk-Sensitive RL
  5. Algorithm
  6. Risk-Sensitive Pessimistic Value Iteration
  7. Variance-Aware RSPVI
  8. Main Results
  9. Theoretical Analysis
  10. Proof Sketch of Theorem \ref{['thm1']}
  11. Proof Sketch of Theorem \ref{['thm2']}
  12. Proof Sketch of Theorem \ref{['thm3']}
  13. Conclusion
  14. Proof of Lemma \ref{['lemma:pessimistic-bound-of-suboptimality']}
  15. Proof of Theorem \ref{['thm1']}
  16. ...and 12 more sections

Key Result

Lemma 3.1

Let $\widehat{Q}$ be a pessimistic estimate of $Q$, satisfying $\mathrm{sign}(\beta)\iota_{\mathrm{exp}, h}(s, a) \geq 0$ for any $(s,a,h)\in {\mathcal{S}}\times\mathcal{A}\times [H]$. $\widehat{V}_{h}(s) = \max_{a \in \mathcal{A}}\widehat{Q}_{h}(s, a)$ denote the value function induced by $\widehat and for all $\beta < 0$, where $\widehat{\pi}$ is the greedy policy taken with respect to $\wideha

Figures (1)

  • Figure 1: Each panel reports the suboptimality of the learned policy from Algorithm \ref{['alg:alg1']} for different $K$ and $h$. $\beta =0.5$ (left) and $\beta =1$ (right). The results are averaged over 20 independent trails, and the mean results are plotted as solid lines. The error bar area corresponds to the $80\%$ confidence interval.

Theorems & Definitions (26)

  • Lemma 3.1
  • Theorem 5.1
  • Theorem 5.2
  • Theorem 5.3
  • proof
  • proof : Proof of Theorem \ref{['thm1']}
  • Lemma 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4
  • ...and 16 more