Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximate Bilevel Difference Convex Programming for Bayesian Risk Markov Decision Processes

Yifan Lin, Enlu Zhou

TL;DR

This work targets infinite-horizon MDPs with unknown parameters and advances BR-MDP as a time-consistent Bayesian framework to mitigate epistemic risk. It casts BR-MDP as a bilevel difference convex program and introduces an offline, approximate algorithm (ABDCP) that solves a sequence of convex subproblems by progressively enriching a finite posterior set, yielding a finite-state controller with provable lower and upper bounds. The approach is validated through offline path-planning and inventory-control experiments, showing improved robustness to parameter uncertainty over risk-neutral and distributionally robust baselines while remaining computationally feasible. The results demonstrate a practical route to risk-aware planning under parameter uncertainty with theoretical performance guarantees and actionable offline policies.

Abstract

We consider infinite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we utilize the recently proposed formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter (or epistemic) uncertainty in MDPs. To solve the infinite-horizon BR-MDP with a class of convex risk measures, we propose a computationally efficient approach called approximate bilevel difference convex programming (ABDCP). The optimization is performed offline and produces the optimal policy that is represented as a finite state controller with desirable performance guarantees. We also demonstrate the empirical performance of the BR-MDP formulation and the proposed algorithm.

Approximate Bilevel Difference Convex Programming for Bayesian Risk Markov Decision Processes

TL;DR

This work targets infinite-horizon MDPs with unknown parameters and advances BR-MDP as a time-consistent Bayesian framework to mitigate epistemic risk. It casts BR-MDP as a bilevel difference convex program and introduces an offline, approximate algorithm (ABDCP) that solves a sequence of convex subproblems by progressively enriching a finite posterior set, yielding a finite-state controller with provable lower and upper bounds. The approach is validated through offline path-planning and inventory-control experiments, showing improved robustness to parameter uncertainty over risk-neutral and distributionally robust baselines while remaining computationally feasible. The results demonstrate a practical route to risk-aware planning under parameter uncertainty with theoretical performance guarantees and actionable offline policies.

Abstract

We consider infinite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we utilize the recently proposed formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter (or epistemic) uncertainty in MDPs. To solve the infinite-horizon BR-MDP with a class of convex risk measures, we propose a computationally efficient approach called approximate bilevel difference convex programming (ABDCP). The optimization is performed offline and produces the optimal policy that is represented as a finite state controller with desirable performance guarantees. We also demonstrate the empirical performance of the BR-MDP formulation and the proposed algorithm.
Paper Structure (21 sections, 7 theorems, 37 equations, 2 figures, 4 tables, 3 algorithms)

This paper contains 21 sections, 7 theorems, 37 equations, 2 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Related Literature
  4. Preliminary: Bayesian Risk MDPs
  5. Preliminary: Risk Measure
  6. Algorithm and Analysis
  7. Bellman Equation and Optimality
  8. Bilevel Difference Convex Programming
  9. Approximate Bilevel Difference Convex Programming
  10. Numerical Experiments
  11. Conclusion
  12. Acknowledgments
  13. Bayes Optimality and Offline Reinforcement Learning
  14. ADDITIONAL LITERATURE REVIEW ON ROBUST MDPS
  15. Technical Proof
  16. ...and 6 more sections

Key Result

Lemma 3.2

The operators $\mathcal{T}^{\pi}$ and $\mathcal{T}$ are monotonic, in the sense that $V \leq V'$ implies $\mathcal{T}^{\pi} V \leq \mathcal{T}^{\pi} V'$ and $\mathcal{T} V \leq \mathcal{T} V'$.

Figures (2)

  • Figure 1: Histogram of the actual performance over 200 replications for different algorithms. Number of data points $N=10$.
  • Figure 2: Path planning terrain map. Colors indicate the road types as follows--blue: highway, red: main road, orange: street, green: lane.

Theorems & Definitions (14)

  • Definition 3.1: Bellman Operator
  • Lemma 3.2: Monotonicity
  • Lemma 3.3: Contraction Mapping
  • Proposition 3.4
  • Corollary 3.5
  • Theorem 3.6
  • Theorem 3.7
  • Theorem A.2
  • proof
  • proof : Proof of Lemma \ref{['Monotonicity']}
  • ...and 4 more