Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk
Sumedh Gupte, Shrey Rakeshkumar Patel, Soumen Pachal, Prashanth L. A., Sanjay P. Bhat
TL;DR
This work addresses risk-sensitive decision making in reinforcement learning by optimizing three convex risk measures—expectiles, utility-based shortfall risk (UBSR), and optimized certainty equivalents (OCE)—within finite-horizon MDPs. It develops policy-gradient theorems for each risk, constructs trajectory-based gradient estimators with non-asymptotic error bounds, and proves smoothness and convergence properties of a general risk-aware policy-gradient framework. The paper also introduces a practical RAPG algorithm and provides non-asymptotic convergence guarantees, then validates the theory with MuJoCo Reacher experiments showing improved performance and reduced variance compared to standard REINFORCE. Overall, it offers a unified, theoretically grounded methodology for risk-aware RL that covers multiple risk measures and demonstrates tangible gains on benchmark tasks.
Abstract
We propose risk-sensitive reinforcement learning algorithms catering to three families of risk measures, namely expectiles, utility-based shortfall risk and optimized certainty equivalent risk. For each risk measure, in the context of a finite horizon Markov decision process, we first derive a policy gradient theorem. Second, we propose estimators of the risk-sensitive policy gradient for each of the aforementioned risk measures, and establish $\mathcal{O}\left(1/m\right)$ mean-squared error bounds for our estimators, where $m$ is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the risk-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to validate the theoretical findings on popular RL benchmarks.