Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning
Shanyu Han, Yangbo He, Yang Liu
TL;DR
The work develops a robust risk-sensitive RL framework based on double-layered risk measures, separating inner cost risk from outer transition-uncertainty risk and embedding this in a measure-theoretic RS-RMDP. It then implements Bayesian Dynamic Programming that alternates posterior transition updates with Bellman updates, estimating the risk-based Bellman operator through Monte Carlo sampling and convex optimization, with strong guarantees on estimator consistency and posterior convergence. Theoretical results establish contraction of the risk-aware Bellman operators, error bounds under model misspecification, and convergence of the on-policy algorithm, along with explicit polynomial sample and computational complexities under Dirichlet priors and CVaR. Empirical validation on synthetic tasks shows favorable risk-sensitivity behavior, robustness to transition perturbations, and convergence advantages, complemented by an option hedging application that outperforms classical benchmarks. Overall, the framework generalizes many RL approaches and provides a principled balance between optimality and robustness under model uncertainty.
Abstract
We propose a novel framework for risk-sensitive reinforcement learning (RSRL) that incorporates robustness against transition uncertainty. We define two distinct yet coupled risk measures: an inner risk measure addressing state and cost randomness and an outer risk measure capturing transition dynamics uncertainty. Our framework unifies and generalizes most existing RL frameworks by permitting general coherent risk measures for both inner and outer risk measures. Within this framework, we construct a risk-sensitive robust Markov decision process (RSRMDP), derive its Bellman equation, and provide error analysis under a given posterior distribution. We further develop a Bayesian Dynamic Programming (Bayesian DP) algorithm that alternates between posterior updates and value iteration. The approach employs an estimator for the risk-based Bellman operator that combines Monte Carlo sampling with convex optimization, for which we prove strong consistency guarantees. Furthermore, we demonstrate that the algorithm converges to a near-optimal policy in the training environment and analyze both the sample complexity and the computational complexity under the Dirichlet posterior and CVaR. Finally, we validate our approach through two numerical experiments. The results exhibit excellent convergence properties while providing intuitive demonstrations of its advantages in both risk-sensitivity and robustness. Empirically, we further demonstrate the advantages of the proposed algorithm through an application on option hedging.