Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning

Xihong Su

TL;DR

This work tackles uncertainty and risk in reinforcement learning by (i) solving multi-model MDPs with Coordinate Ascent Dynamic Programming (CADP) that adapts model weights to guarantee monotone policy improvement, (ii) establishing contraction conditions for exponential risk-averse total-reward criteria (ERM-TRC and EVaR-TRC) and providing DP/LP algorithms to compute optimal stationary policies in transient MDPs, and (iii) delivering model-free Q-learning algorithms for ERM-TRC and EVaR-TRC that converge despite potential non-contraction by leveraging operator monotonicity and elicitability. CADP bridges policy gradient and DP, outperforming prior DP methods on benchmark MMDPs while remaining tractable in tabular settings. For risk-averse TRCs, the work proves the existence of optimal stationary policies under ERM/EVaR, with DP algorithms that extend beyond discounting and apply to both positive and negative rewards. The model-free Q-learning extensions rely on the elicitability of ERM and careful analysis of the exponential Bellman operator, achieving convergence to optimal risk-averse value functions and enabling practical learning of risk-sensitive policies without full model knowledge. Collectively, these results advance reliable, risk-aware RL for uncertain environments and set the stage for scalable extensions to deep RL and broader risk measures.

Abstract

This dissertation makes three main contributions. First, We identify a new connection between policy gradient and dynamic programming in MMDPs and propose the Coordinate Ascent Dynamic Programming (CADP) algorithm to compute a Markov policy that maximizes the discounted return averaged over the uncertain models. CADP adjusts model weights iteratively to guarantee monotone policy improvements to a local maximum. Second, We establish sufficient and necessary conditions for the exponential ERM Bellman operator to be a contraction and prove the existence of stationary deterministic optimal policies for ERM-TRC and EVaR-TRC. We also propose exponential value iteration, policy iteration, and linear programming algorithms for computing optimal stationary policies for ERM-TRC and EVaR-TRC. Third, We propose model-free Q-learning algorithms for computing policies with risk-averse objectives: ERM-TRC and EVaR-TRC. The challenge is that Q-learning ERM Bellman may not be a contraction. Instead, we use the monotonicity of Q-learning ERM Bellman operators to derive a rigorous proof that the ERM-TRC and the EVaR-TRC Q-learning algorithms converge to the optimal risk-averse value functions. The proposed Q-learning algorithms compute the optimal stationary policy for ERM-TRC and EVaR-TRC.

Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning

TL;DR

This work tackles uncertainty and risk in reinforcement learning by (i) solving multi-model MDPs with Coordinate Ascent Dynamic Programming (CADP) that adapts model weights to guarantee monotone policy improvement, (ii) establishing contraction conditions for exponential risk-averse total-reward criteria (ERM-TRC and EVaR-TRC) and providing DP/LP algorithms to compute optimal stationary policies in transient MDPs, and (iii) delivering model-free Q-learning algorithms for ERM-TRC and EVaR-TRC that converge despite potential non-contraction by leveraging operator monotonicity and elicitability. CADP bridges policy gradient and DP, outperforming prior DP methods on benchmark MMDPs while remaining tractable in tabular settings. For risk-averse TRCs, the work proves the existence of optimal stationary policies under ERM/EVaR, with DP algorithms that extend beyond discounting and apply to both positive and negative rewards. The model-free Q-learning extensions rely on the elicitability of ERM and careful analysis of the exponential Bellman operator, achieving convergence to optimal risk-averse value functions and enabling practical learning of risk-sensitive policies without full model knowledge. Collectively, these results advance reliable, risk-aware RL for uncertain environments and set the stage for scalable extensions to deep RL and broader risk measures.

Abstract

This dissertation makes three main contributions. First, We identify a new connection between policy gradient and dynamic programming in MMDPs and propose the Coordinate Ascent Dynamic Programming (CADP) algorithm to compute a Markov policy that maximizes the discounted return averaged over the uncertain models. CADP adjusts model weights iteratively to guarantee monotone policy improvements to a local maximum. Second, We establish sufficient and necessary conditions for the exponential ERM Bellman operator to be a contraction and prove the existence of stationary deterministic optimal policies for ERM-TRC and EVaR-TRC. We also propose exponential value iteration, policy iteration, and linear programming algorithms for computing optimal stationary policies for ERM-TRC and EVaR-TRC. Third, We propose model-free Q-learning algorithms for computing policies with risk-averse objectives: ERM-TRC and EVaR-TRC. The challenge is that Q-learning ERM Bellman may not be a contraction. Instead, we use the monotonicity of Q-learning ERM Bellman operators to derive a rigorous proof that the ERM-TRC and the EVaR-TRC Q-learning algorithms converge to the optimal risk-averse value functions. The proposed Q-learning algorithms compute the optimal stationary policy for ERM-TRC and EVaR-TRC.
Paper Structure (64 sections, 50 theorems, 205 equations, 13 figures, 7 tables, 3 algorithms)

This paper contains 64 sections, 50 theorems, 205 equations, 13 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Mitigating the Effects of Epistemic Uncertainty
  3. Mitigating the Effects of Aleatoric Uncertainty and Risk
  4. Model-based Risk-averse Policy
  5. Model-free Risk-averse Policy
  6. Overview of Dissertation
  7. Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programming
  8. Introduction
  9. Related Work
  10. Robust and soft-robust MDPs
  11. Multi-model MDPs
  12. POMDPs
  13. Bayesian Multi-armed Bandits
  14. Policy Gradient
  15. Framework: Multi-model MDPs
  16. ...and 49 more sections

Key Result

Theorem 2.4.1

The gradient of $\rho$ defined in eq:return for each $t \in \mathcal{T}$, $\hat{s}\in \mathcal{S}$, $\hat{a} \in \mathcal{A}$, and $\pi \in \Pi_{\mathrm{R}}$ satisfies that where $q$ and $b$ are defined in eq:value-0f-(state-action) and eq:belief-update respectively.

Figures (13)

  • Figure 1: Left: model $m_{1}$, right: model $m_{2}$
  • Figure 2: Mean Returns of CADP with Different Initial Policies at Different Iterations
  • Figure 3: Left: a discounted MDP, Right: a transient MDP
  • Figure 4: ERM values with TRC and discounted criteria.
  • Figure 5: The optimal EVaR-TRC policies.
  • ...and 8 more figures

Theorems & Definitions (92)

  • Definition 2.4.1
  • Theorem 2.4.1
  • proof
  • Corollary 2.4.2
  • Corollary 2.4.2
  • proof
  • Theorem 2.5.1
  • proof
  • Corollary 2.5.2
  • Theorem 2.5.3
  • ...and 82 more