Semismooth Newton Methods for Risk-Averse Markov Decision Processes
Matilde Gargiani, Francesco Micheli, Anastasios Tsiamis, John Lygeros
TL;DR
This work tackles the challenge of solving risk-averse Markov Decision Processes (MDPs) governed by Markovian coherent risk measures in discounted finite-horizon settings. It introduces a semismooth Newton framework that recasts the risk-averse Bellman equations as a nonsmooth root-finding problem and designs three Newton-based methods (SNMI, SNMII, SNMIII) with convergence guarantees. The analysis unifies risk-averse policy iteration with semismooth Newton theory, proving global convergence and, under suitable structure (e.g., PC$^1$ or CVaR), local Q-quadratic or superlinear convergence. Numerical experiments on CVaR-MDPs show competitive performance and highlight the advantages of the proposed Newton-based approaches over risk-averse value iteration, while enabling parallelizable convex subproblems. Overall, the paper provides a rigorous, versatile framework for efficiently solving risk-averse DP problems and clarifies the convergence benefits of policy-iteration schemes in this setting.
Abstract
Inspired by semismooth Newton methods, we propose a general framework for designing solution methods with convergence guarantees for risk-averse Markov decision processes. Our approach accommodates a wide variety of risk measures by leveraging the assumption of Markovian coherent risk measures. To demonstrate the versatility and effectiveness of this framework, we design three distinct solution methods, each with proven convergence guarantees and competitive empirical performance. Validation results on benchmark problems demonstrate the competitive performance of our methods. Furthermore, we establish that risk-averse policy iteration can be interpreted as an instance of semismooth Newton's method. This insight explains its superior convergence properties compared to risk-averse value iteration. The core contribution of our work, however, lies in developing an algorithmic framework inspired by semismooth Newton methods, rather than evaluating specific risk measures or advocating for risk-averse approaches over risk-neutral ones in particular applications.