Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation
Tonghe Zhang, Yu Chen, Longbo Huang
TL;DR
This paper addresses risk-sensitive reinforcement learning in partially observable environments with hindsight observations, framing the objective via the entropic risk measure $J(\pi;\mathcal{P},\gamma)=\frac{1}{\gamma}\log\mathbb{E}_{\mathcal{P}}^{\pi}[e^{\gamma\sum_t r_t}]$ and proving a polynomial regret bound for a new algorithm. It introduces hindsight into the POMDP setting and leverages a Change-of-Measure technique to define a reference model $\mathcal{P}'$, along with risk Beliefs, Conjugate Beliefs, and Beta vectors to recover a tractable, Markovian structure for planning. The Beta Vector Value Iteration (BVVI) algorithm uses a risk-aware bonus $\mathsf{b}_h^k$ and optimistic beta estimates to achieve near-optimal sample complexity, with regret that scales as $\tilde{O}\left( \frac{e^{|\gamma|H}-1}{|\gamma|H} H^{5/2} \sqrt{K S^2 A O} \right)$ up to logarithmic factors, recovering improved or matching bounds in risk-neutral or fully observable limits. The work advances theoretical understanding of risk-sensitive RL under partial observability and hindsight, offering a solid foundation for future extensions to function approximation and alternative risk criteria.
Abstract
This work pioneers regret analysis of risk-sensitive reinforcement learning in partially observable environments with hindsight observation, addressing a gap in theoretical exploration. We introduce a novel formulation that integrates hindsight observations into a Partially Observable Markov Decision Process (POMDP) framework, where the goal is to optimize accumulated reward under the entropic risk measure. We develop the first provably efficient RL algorithm tailored for this setting. We also prove by rigorous analysis that our algorithm achieves polynomial regret $\tilde{O}\left(\frac{e^{|γ|H}-1}{|γ|H}H^2\sqrt{KHS^2OA}\right)$, which outperforms or matches existing upper bounds when the model degenerates to risk-neutral or fully observable settings. We adopt the method of change-of-measure and develop a novel analytical tool of beta vectors to streamline mathematical derivations. These techniques are of particular interest to the theoretical study of reinforcement learning.