Policy stability and ultimate stationarity in discounted risk-sensitive stochastic control

TL;DR

This paper analyzes discounted risk-sensitive MDPs on finite state-action spaces under entropic utility, focusing on the stability of optimal policies and value functions as discounting and risk aversion vary. It establishes that optimal policies are ultimately stationary for any $\beta\in(0,1)$ and $\gamma\neq0$, and that stationary optimality emerges near $\gamma=0$; it also links discounted problems to the averaged formulation via mixing, via Blackwell-type results, and studies the vanishing-discount and vanishing-risk-sensitivity limits, including span-contraction properties and moment-optimality. The results provide structural insights into how risk sensitivity and discounting shape policy non-stationarity, tail behavior (turnpikes), and the connection to long-run averaged criteria, supported by numerical demonstrations of non-stationary discounted policies. Together, they inform both theory and reinforcement learning practice by clarifying when stationary behavior can be expected and how limits relate different long-run objectives.

Abstract

We study discrete-time Markov Decision Processes (MDPs) on finite state-action spaces and analyze the stability of optimal policies and value functions in the long-run discounted risk-sensitive objective setting. Our analysis addresses robustness with respect to perturbations of the risk-aversion parameter and the discount factor, the emergence of ultimate stationarity, and the interaction between discounted and averaged formulations under suitable mixing assumptions. We further investigate limiting regimes associated with vanishing discount and vanishing risk sensitivity, and discuss the role of Blackwell-type stability properties in the discounted setting. Finally, we provide numerical illustrations that highlight the intrinsic non-stationarity of optimal discounted risk-sensitive policies.

Figures (4)

• Figure 1: The plot presents the (ultimately stationary) optimal policy turnpike $N(\beta,\gamma)$ for different values of $\gamma\in[-2.5,2.5]$ and $\beta\in(0.9,0.995)$ under the setup in Example \ref{['ex:2']}, for $R=7$. The vertical dashed lined mark $\gamma$-based thresholds when the averaged per unit of time optimal (stationary) policy is changing. As one can see, we can observe that when $\gamma\to 0$, then the turnpike approaches zero leading to stationary optimal policy. On the other hand, when $\beta\to 1$, the turnpike could approach either 0 or $+\infty$ depending on the limit policy for the averaged problem.
• Figure 2: The plot presents the value function for the stationary policies in the limit case when $\beta\to 1$ (risk-averaged) and when $\gamma\to 0$ (risk-neutral) using the value functions \ref{['eq:J.averaged.lottery']} and \ref{['eq:J.neutral.lottery']}; we have multiplied the second function by $(1-\beta)$ to normalize the output as otherwise it tends to $+\infty$ as $\beta\to 1$. The results are presented for different values of $\gamma\in[-2.5,2.5]$ and $\beta\in(0.9,0.995)$ under the setup in Example \ref{['ex:2']}, for $R=7$. The vertical dashed lined mark $\gamma$-based thresholds when the averaged per unit of time optimal (stationary) policy is changing. Note that in the considered set, the policy $u_a$ is risk-neutral discounted optimal for all values of $\beta\in [0.9,0.995]$.
• Figure 3: The plot presents the value function for the stationary policies in the limit case when $\beta\to 1$ (risk-averaged) and when $\gamma\to 0$ (risk-neutral) for the decreased reward $R=3.5$ in the Example \ref{['ex:2']} framework; see Figure \ref{['F:ex2b']} caption for detailed description. Note that the risk-neutral optimal discounted policy is changing - the switch is at $\beta\approx 0.9525$.
• Figure 4: The plot presents the (ultimately stationary) optimal policy turnpike $N(\beta,\gamma)$ for different values of $\gamma\in[-2.5,2.5]$ and $\beta\in(0.9,0.995)$ under the setup in Example \ref{['ex:3']}, with decreased reward $R=3.5$; see Figure \ref{['F:ex2']} caption for detailed description. Horizontal dashed line indicate optimal stationary risk-neutral discounted policy switch point.

Theorems & Definitions (39)

• Proposition 2.1: Properties of the policy value function
• Proof 1
• Theorem 3.1
• Proof 2
• Proposition 3.2: General Properties of the optimal discounted policies and value functions
• Proof 3
• Theorem 4.1: Existence of an optimal stationary discounted policy for a small risk aversion
• Proof 4
• Corollary 4.2
• Proof 5